2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · 2022 · 345 citations · 28 references
ComputingComputer ScienceData MiningMachine LearningPrivacy EngineeringData IntegrationEnsemble LearningEnsemble MethodsFederated LearningGenerative ModelingMachine Learning For Big DataAggregationConvergenceHeterogeneityHeterogeneity ModelingKnowledge DistillationModel AggregationPrivacyPrivacy-preserving Model Updates
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. Data heterogeneity is one of the main challenges in FL, which results in slow convergence and degraded performance. Most existing approaches only tackle the heterogeneity challenge by restricting the local model update in client, ignoring the performance drop caused by direct global model aggregation. Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation. Concretely, FedFTG explores the input space of local models through a generator, and uses it to transfer the knowledge from local models to the global model. Besides, we propose a hard sample mining scheme to achieve effective knowledge distillation throughout the training. In addition, we develop customized label sampling and class-level ensemble to derive maximum utilization of knowledge, which implicitly mitigates the distribution discrepancy across clients. Extensive experiments show that our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
28
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Model-Contrastive Federated Learning
Qinbin Li, Bingsheng He, Dawn Song · 2021 · 1.4K citations
Federated Learning for Mobile Keyboard Prediction
Andrew Hard, K. Praveen Kumar Rao, Rajiv Mathews et al. · arXiv (Cornell University) · 2018 · 1.1K citations · Full text
On the Convergence of FedAvg on Non-IID Data
Xiang Li, Kaixuan Huang, Wenhao Yang et al. · arXiv (Cornell University) · 2019 · 1K citations · Full text
A large-scale car dataset for fine-grained categorization and verification
Linjie Yang, Ping Luo, Chen Change Loy et al. · 2015 · 844 citations