Communication-efficient federated learning via knowledge distillation

TLDR

Federated learning trains models on decentralized data by exchanging local updates, but large model updates and many communication rounds create high overhead and environmental impact. The study proposes FedKD, a federated learning method aimed at reducing communication costs while maintaining performance. FedKD achieves this through adaptive mutual knowledge distillation and dynamic gradient compression. FedKD reduces communication cost by up to 94.89% and achieves results comparable to centralized learning, demonstrating its potential for privacy‑preserving intelligent systems in healthcare and personalization.

Abstract

Federated learning is a privacy-preserving machine learning technique to train intelligent models from decentralized data, which enables exploiting private data by communicating local model updates in each iteration of model learning rather than the raw data. However, model updates can be extremely large if they contain numerous parameters, and many rounds of communication are needed for model training. The huge communication cost in federated learning leads to heavy overheads on clients and high environmental burdens. Here, we present a federated learning method named FedKD that is both communication-efficient and effective, based on adaptive mutual knowledge distillation and dynamic gradient compression techniques. FedKD is validated on three different scenarios that need privacy protection, showing that it maximally can reduce 94.89% of communication cost and achieve competitive results with centralized model learning. FedKD provides a potential to efficiently deploy privacy-preserving intelligent systems in many scenarios, such as intelligent healthcare and personalization.