IEEE Signal Processing Magazine · 2020 · 121 citations · 47 references
Natural SciencesComputer ScienceDistributed SystemsMachine LearningOptimizationConvex OptimizationGradient-based OptimizationOptimization For Machine LearningRobust OptimizationStochastic OptimizationConvergenceFirst-order OptimizationRobust PerformanceSmooth OptimizationStochastic Gradient Descent
Decentralized methods to solve finite-sum minimization problems are important in many signal processing and machine learning tasks where the data samples are distributed across a network of nodes, and raw data sharing is not permitted due to privacy and/or resource constraints. In this article, we review decentralized stochastic first-order methods and provide a unified algorithmic framework that combines variance reduction with gradient tracking to achieve robust performance and fast convergence. We provide explicit theoretical guarantees of the corresponding methods when the objective functions are smooth and strongly convex and show their applicability to nonconvex problems via numerical experiments. Throughout the article, we provide intuitive illustrations of the main technical ideas by casting appropriate tradeoffs and comparisons among the methods of interest and by highlighting applications to decentralized training of machine learning models.
47
Pattern Recognition and Machine Learning
Journal of Electronic Imaging · 2007 · 22.1K citations
Consensus and Cooperation in Networked Multi-Agent Systems
Reza Olfati‐Saber, J.A. Fax, Richard M. Murray · Proceedings of the IEEE · 2007 · 10.3K citations
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan, Eider Moore, Daniel Ramage et al. · arXiv (Cornell University) · 2016 · 5.2K citations · Full text