2020 · 24 citations · 31 references
ComputingComputer ScienceComputer SystemsDistributed SystemsEdge ComputingMachine LearningDistributed AlgorithmsDistributed ComputingDistributed Data ProcessingEdge Artificial IntelligenceEdge ReliabilityMobile Edge ComputingAggregationClient-server CommunicationDistributed Training At The EdgeHierarchical Edge ArchitectureHierarchical Edge ComputingModel AggregationPrivacy
Client devices at the edge are generating increasingly large amounts of rich data suitable for learning powerful statistical models. However, privacy concerns and heavy communication load make it infeasible to move the client data to a centralized location for training. In many distributed learning setups, client nodes carry out gradient computations on their local data while the central master server receives the local gradients and aggregates them to take the global model update step. To guarantee robustness against straggling communication links, we consider a hierarchical setup with n <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</sub> clients and n <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">h</sub> reliable helper nodes that are available to aid in gradient aggregation at the master. To achieve resiliency against straggling client-to-helpers links, we propose two approaches leveraging coded redundancy. First is the Aligned Repetition Coding (ARC) that repeats gradient components on the helper links, allowing significant partial aggregations at the helpers, resulting in a helpers-to-master communication load (C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HM</sub> ) of O(n <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">h</sub> ). ARC however results in a client-to-helpers communication load (C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EH</sub> ) of Θ(n <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">h</sub> ), which is prohibitive for client nodes due to limited and costly bandwidth. We thus propose Aligned Minimum Distance Separable Coding (AMC) that achieves optimal C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EH</sub> of Θ(1) for a given resiliency threshold by using MDS code over the gradient components, while achieving a C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HM</sub> of O(n <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</sub> ).
31
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan, Eider Moore, Daniel Ramage et al. · arXiv (Cornell University) · 2016 · 5.2K citations · Full text
The Emergence of Edge Computing
Mahadev Satyanarayanan · Computer · 2017 · 2.5K citations
Speeding Up Distributed Machine Learning Using Codes
Kangwook Lee, Maximilian Lam, Ramtin Pedarsani et al. · IEEE Transactions on Information Theory · 2017 · 866 citations · Full text