FedUR: Federated Learning Optimization Through Adaptive Centralized Learning Optimizers

Abstract

Introducing adaptiveness to federated learning has recently ushered in a new way to optimize its convergence performance. However, adaptive learning strategies originally designed in centralized machine learning are in naїve extended to federated learning in existing works, which does not necessarily improve convergence performance and further reduce communication overhead as expected. In this paper, we fully investigate those centralized learning-based adaptive learning strategies, and propose an adaptive <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fed</u> erated learning algorithm targeting the model parameter <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</u> pdate <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</u> ule, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FedUR</i> . Convergence upper bounds under <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FedUR</i> are derived from the aspect of both local iterations and global aggregations. Through comparison with the convergence upper bounds of original federated learning, we theoretically analyze how those strategies should be tuned to help federated learning effectively optimize convergence performance and reduce overall communication overhead. Extensive experiments are conducted based on several real datasets and machine learning models, which show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FedUR</i> can effectively increase final convergence accuracy with even lower communication overhead requirement.