IEEE Transactions on Control of Network Systems · 2023 · 10 citations · 26 references
ComputingComputer ScienceDistributed SystemsMachine LearningOptimizationConvex OptimizationDistributed AlgorithmsDistributed ComputingGradient-based OptimizationNeural NetworksStochastic OptimizationAsynchronous Distributed AlgorithmsConvergence Rate AnalysisConvolutional Neural NetworksDistributed Machine LearningDistributed OptimizationMinimizationNonconvex OptimizationStochastic Gradient Descent
We consider a decentralized stochastic optimization problem over a network of agents, modeled as a directed graph: Agents aim to asynchronously minimize the average of their individual losses (possibly non-convex), each one having access only to a noisy estimate of the gradient of its own function. We propose an asynchronous distributed algorithm for such a class of problems. The algorithm combines stochastic gradients with tracking in an asynchronous push-sum framework and obtains a sublinear convergence rate, matching the rate of the centralized stochastic gradient descent applied to the nonconvex minimization. Our experiments on a non-convex image classification task using convolutional neural network validate the convergence of our proposed algorithm across different number of nodes and graph connectivity percentages.
26
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan, Eider Moore, Daniel Ramage et al. · arXiv (Cornell University) · 2016 · 5.2K citations · Full text
Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
Xiangru Lian, Yijun Huang, Yuncheng Li et al. · arXiv (Cornell University) · 2015 · 238 citations · Full text