Think Locally, Act Globally: Federated Learning with Local and Global\n Representations

Abstract

Federated learning is a method of training models on private data distributed\nover multiple devices. To keep device data private, the global model is trained\nby only communicating parameters and updates which poses scalability challenges\nfor large models. To this end, we propose a new federated learning algorithm\nthat jointly learns compact local representations on each device and a global\nmodel across all devices. As a result, the global model can be smaller since it\nonly operates on local representations, reducing the number of communicated\nparameters. Theoretically, we provide a generalization analysis which shows\nthat a combination of local and global models reduces both variance in the data\nas well as variance across device distributions. Empirically, we demonstrate\nthat local models enable communication-efficient training while retaining\nperformance. We also evaluate on the task of personalized mood prediction from\nreal-world mobile data where privacy is key. Finally, local models handle\nheterogeneous data from new devices, and learn fair representations that\nobfuscate protected attributes such as race, age, and gender.\n