2023 · 42 citations · 21 references
ComputingComputer ScienceComputer SystemsHeterogeneous ComputingMachine LearningReal-time SystemsAdaptive LearningAsynchronous LearningDistributed LearningFederated LearningAsynchronous Federated LearningConvergence Rate AnalysisCross-device Federated LearningIntermittent Connectivity HandlingModel AggregationReal-time Resource AvailabilityResource Sharing In Real-time SystemsTraining
In cross-device Federated Learning (FL) environments, scaling synchronous FL methods is challenging as stragglers hinder the training process. Moreover, the availability of each client to join the training is highly variable over time due to system heterogeneities and intermittent connectivity. Recent asynchronous FL methods (e.g., FedBuff [22]) have been proposed to overcome these issues by allowing slower users to continue their work on local training based on stale models and to contribute to aggregation when ready. However, we show empirically that this method can lead to a substantial drop in training accuracy as well as a slower convergence rate. The primary reason is that fast-speed devices contribute to many more rounds of aggregation while others join more intermittently or not at all, and with stale model updates. To overcome this barrier, we propose TimelyFL, a heterogeneity-aware asynchronous FL framework with adaptive partial training. During the training, TimelyFL adjusts the local training workload based on the real-time resource capabilities of each client, aiming to allow more available clients to join in the global update without staleness. We demonstrate the performance benefits of TimelyFL by conducting extensive experiments on various datasets (e.g., CIFAR-10, Google Speech, and Reddit) and models (e.g., ResNet20, VGG11, and ALBERT). In comparison with the state-of-the-art (i.e., FedBuff), our evaluations reveal that TimelyFL improves participation rate by 21.13%, harvests 1.28× - 2.89× more efficiency on convergence rate, and provides a 6.25% increment on test accuracy.
21
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Advances and Open Problems in Federated Learning
Peter Kairouz, H. Brendan McMahan, Brendan Avent et al. · Foundations and Trends® in Machine Learning · 2020 · 4.7K citations · Full text
Google Speech Commands-Musan test set
arXiv (Cornell University) · 2018 · 1.1K citations · Full text
Computing, Data Science, Acoustics +14
Asynchronous Federated Optimization
Cong Xie, Indranil Gupta · arXiv (Cornell University) · 2019 · 442 citations · Full text