Federated Learning: Strategies for Improving Communication Efficiency

TLDR

Federated Learning trains a centralized model from data distributed across many clients with unreliable, slow connections, making communication efficiency critical, especially for mobile phone clients. The study aims to reduce uplink communication costs in federated learning by proposing structured updates and sketched updates. Clients independently compute model updates from local data and send them to a central server for aggregation; the proposed methods use structured updates (low‑rank or random masks) and sketched updates (quantization, random rotations, subsampling) to compress the updates. Experiments on convolutional and recurrent networks show a two‑order‑of‑magnitude reduction in communication cost.

Abstract

Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model. The typical clients in this setting are mobile phones, and communication efficiency is of the utmost importance. In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, where we learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling before sending it to the server. Experiments on both convolutional and recurrent networks show that the proposed methods can reduce the communication cost by two orders of magnitude.