IEEE Journal on Selected Areas in Information Theory · 2022 · 24 citations · 23 references
ComputingComputer ScienceDistributed SystemsMachine LearningMobile ComputingDistributed AlgorithmsDistributed ComputingDistributed LearningFederated LearningPeer-to-peer NetworksPeer-to-peer SystemsAggregationBayesian NetworksBayesian Variational InferenceDecentralized Federated LearningModel AggregationPeer-to-peer Network ArchitectureVariational BayesVariational Bayesian Inference
This paper proposes a federated supervised learning framework over a general peer-to-peer network with agents that act in a variational Bayesian fashion. The proposed framework consists of local agents where each of which keeps a local “posterior probability distribution” over the parameters of a global model; the updating of the posterior over time happens in a local fashion according to two subroutines of: 1) variational model training given (a batch of) local labeled data, and 2) asynchronous communication and model aggregation with the 1-hop neighbors. Inspired by the popular federated learning (model averaging), the framework allows the training data to remain distributed on mobile devices while utilizing a peer-to-peer model aggregation in a social network. The proposed framework is shown to allow for a systematic treatment of model aggregation over any arbitrary connected graph with consistent (in general, non-iid) local labeled data. Specifically, under mild technical conditions, the proposed algorithm allows agents with local data to learn a shared model explaining the global training data in a decentralized fashion over an arbitrary peering/connectivity graph. Furthermore, the rate of convergence is characterized and shown to be a function of each individual agent’s data quality weighted by its eigenvector centrality. Empirically, the proposed methodology is shown to work well with efficient variation Bayesian inference techniques to train Bayesian neural networks in a decentralized manner even when the local data batches are not identically distributed.
23
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Federated Learning: Strategies for Improving Communication Efficiency
Jakub Konečný, H. Brendan McMahan, Felix X. Yu et al. · arXiv (Cornell University) · 2016 · 3K citations · Full text
Fast linear iterations for distributed averaging
Lin Xiao, Stephen Boyd · Systems & Control Letters · 2004 · 2.8K citations