Proceedings of the AAAI Conference on Artificial Intelligence · 2022 · 87 citations · 40 references
ComputingComputer ScienceDistributed SystemsMachine LearningPrivacy EngineeringData PrivacyData ProtectionDifferential PrivacyFederated LearningFederated Learning PrivacyScalable Machine LearningData DistributionKnowledge DistillationPrivacy LeakagePrivacy PreservationPrivacy RiskPrivacy-preserving LearningPrivacy-preserving Machine LearningText Classification
Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model while the training data remains decentralized. Existing FL methods typically share model parameters or employ co-distillation to address the issue of unbalanced data distribution. However, they suffer from communication bottlenecks. More importantly, they risk privacy leakage risk. In this work, we develop a privacy preserving and communication efficient method in a FL framework with one-shot offline knowledge distillation using unlabeled, cross-domain, non-sensitive public data. We propose a quantized and noisy ensemble of local predictions from completely trained local models for stronger privacy guarantees without sacrificing accuracy. Based on extensive experiments on image classification and text classification tasks, we show that our method outperforms baseline FL algorithms with superior performance in both accuracy and data privacy preservation.
40
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, Oriol Vinyals · arXiv (Cornell University) · 2015 · 14K citations · Full text
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Richard Socher, Alex Perelygin, Jean Y. Wu et al. · 2013 · 6.8K citations · Full text
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan, Eider Moore, Daniel Ramage et al. · arXiv (Cornell University) · 2016 · 5.2K citations · Full text