2024 · 13 citations · 14 references
ComputingComputer EngineeringComputer ScienceHeterogeneous ComputingMachine LearningParallel ComputingData ParallelismDeep LearningDistributed ComputingHeterogeneous Cluster ComputingHeterogeneous System ArchitectureOptimization For Machine LearningAutomated Program SynthesisCommunication OptimizationCommunication PrimitivesConvolutional Neural NetworksHeterogeneous Cluster ArchitecturePartitioningProgram SynthesisSolution Space Exploration
Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning. This paper presents HAP, an automated system designed to expedite SPMD DNN training on heterogeneous clusters. HAP jointly optimizes the tensor sharding strategy, sharding ratios across heterogeneous devices and the communication methods for tensor exchanges for optimized distributed training with SPMD parallelism. We novelly formulate model partitioning as a program synthesis problem, in which we generate a distributed program from scratch on a distributed instruction set that semantically resembles the program designed for a single device, and systematically explore the solution space with an A-based search algorithm. We derive the optimal tensor sharding ratios by formulating it as a linear programming problem. Additionally, HAP explores tensor communication optimization in a heterogeneous cluster and integrates it as part of the program synthesis process, for automatically choosing optimal collective communication primitives and applying sufficient factor broadcasting technique. Extensive experiments on representative workloads demonstrate that HAP achieves up to 2.41x speed-up on heterogeneous clusters.
14
An axiomatic basis for computer programming
C. A. R. Hoare · Communications of the ACM · 1969 · 3.9K citations · Full text
2017 USENIX Annual Technical Conference (USENIX ATC'17)
Do Le Quoc, Martin S. Beck, Pramod Bhatotia et al. · 2017 · 3.1K citations
Deepak Narayanan, Aaron Harlap, Amar Phanishayee et al. · 2019 · 752 citations
ZeRO: Memory optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase et al. · 2020 · 741 citations
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase et al. · 2020 · 709 citations