Proceedings of the AAAI Conference on Artificial Intelligence · 2019 · 41 citations · 34 references
ComputingComputer EngineeringComputer ScienceCluster ComputingDistributed SystemsMachine LearningDeep LearningDistributed AlgorithmsDistributed ComputingOptimization For Machine LearningScalable Algorithm DesignScalable Machine LearningBatched CommunicationBatching TechniquesCommunication-efficient TrainingDistributed Machine LearningGreedy HeuristicsParameter Server Architecture
Scalability of distributed deep learning (DL) training with parameter server architecture is often communication constrained in large clusters. There are recent efforts that use a layer by layer strategy to overlap gradient communication with backward computation so as to reduce the impact of communication constraint on the scalability. However, the approaches cannot be effectively applied to the overlap between parameter communication and forward computation. In this paper, we propose and design iBatch, a novel communication approach that batches parameter communication and forward computation to overlap them with each other. We formulate the batching decision as an optimization problem and solve it based on greedy algorithm to derive communication and computation batches. We implement iBatch in the open-source DL framework BigDL and perform evaluations with various DL workloads. Experimental results show that iBatch improves the scalability of a cluster of 72 nodes by up to 73% over the default PS and 41% over the layer by layer strategy.
34
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia et al. · 2015 · 46.8K citations
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke et al. · 2017 · 12.6K citations · Full text
TensorFlow: A system for large-scale machine learning
Martı́n Abadi, Paul Barham, Jianmin Chen et al. · arXiv (Cornell University) · 2016 · 8.8K citations · Full text