arXiv (Cornell University) · 2016 · 94 citations · 15 references
ComputingCommunicationComputer ScienceAlgorithmsDistributed SystemsMachine LearningDistributed AlgorithmsDistributed ComputingDistributed CoordinationPeer-to-peer NetworkingPeer-to-peer NetworksPeer-to-peer SystemsDecentralized LearningGossip AlgorithmsLabel PropagationPeer-to-peer ArchitecturesPeer-to-peer Network Architecture
We consider a set of learning agents in a collaborative peer-to-peer network, where each agent learns a personalized model according to its own learning objective. The question addressed in this paper is: how can agents improve upon their locally trained model by communicating with other agents that have similar objectives? We introduce and analyze two asynchronous gossip algorithms running in a fully decentralized manner. Our first approach, inspired from label propagation, aims to smooth pre-trained local models over the network while accounting for the confidence that each agent has in its initial model. In our second approach, agents jointly learn and propagate their model by making iterative updates based on both their local dataset and the behavior of their neighbors. To optimize this challenging objective, our decentralized algorithm is based on ADMM.
15
Rich Caruana · Machine Learning · 1997 · 6.3K citations · Full text
Learning with Local and Global Consistency
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal et al. · MPG.PuRe (Max Planck Society) · 2003 · 3.8K citations · Full text
Stephen Boyd, Aritra Ghosh, Balaji Prabhakar et al. · IEEE Transactions on Information Theory · 2006 · 2.5K citations
Federated Learning of Deep Networks using Model Averaging
H. Brendan McMahan, Eider Moore, Daniel Ramage et al. · arXiv (Cornell University) · 2016 · 984 citations · Full text