Concurrency and Computation Practice and Experience · 2012 · 151 citations · 32 references
ComputingComputer ScienceDistributed SystemsMachine LearningStatistical LearningDistributed AlgorithmsDistributed ComputingDistributed LearningEnsemble LearningEnsemble MethodsPeer-to-peer NetworksDistributed Machine LearningGossip AlgorithmsLinear ModelsWeighted Majority VotingWeighted VotingWeighted Voting Systems
Machine learning on fully distributed data, where each node holds a single record and raw data cannot be shared due to privacy, poses a challenge because local models cannot be learned, reliability guarantees are weak, and communication costs must remain low. The study proposes gossip learning, a generic method in which multiple models perform parallel random walks over the network, learn online, and are combined via ensemble learning, instantiated for linear-model classification. Gossip learning implements parallel random walks of linear models over the network, applies online learning to each model, and aggregates them through ensemble methods, with theoretical convergence guarantees and empirical validation on benchmark datasets. The method achieves a virtual weighted voting over exponentially many models with negligible extra cost, converges theoretically, and demonstrates strong performance and robustness in extensive benchmark experiments. © 2012.
SUMMARY Machine learning over fully distributed data poses an important problem in peer‐to‐peer applications. In this model, we have one data record at each network node but without the possibility to move raw data because of privacy considerations. For example, user profiles, ratings, history, or sensor readings can represent this case. This problem is difficult because there is no possibility to learn local models; the system model offers almost no guarantee for reliability, yet the communication cost needs to be kept low. Here, we propose gossip learning, a generic approach that is based on multiple models taking random walks over the network in parallel, while applying an online learning algorithm to improve themselves, and getting combined via ensemble learning methods. We present an instantiation of this approach for the case of classification with linear models. Our main contribution is an ensemble learning method, which—through the continuous combination of the models in the network—implements a virtual weighted voting mechanism over an exponential number of models at practically no extra cost as compared with independent random walks. We prove the convergence of the method theoretically, and perform extensive experiments on benchmark data sets. Our experimental analysis demonstrates the performance and robustness of the proposed approach. Copyright © 2012 John Wiley & Sons, Ltd.
32
UCI Machine Learning Repository
Arthur Asuncion · Medical Entomology and Zoology · 2007 · 24.4K citations
Leo Breiman · Machine Learning · 1996 · 17K citations · Full text
Leo Breiman · Machine Learning · 1996 · 16.4K citations · Full text
Lior Rokach · Artificial Intelligence Review · 2009 · 2.6K citations · Full text