2017 · 14 citations · 17 references
ComputingComputer ScienceAlgorithmsArtificial IntelligenceMachine LearningDeep LearningGradient-based OptimizationMachine Learning For Big DataNeural NetworksConvergence Rate AnalysisDelay CompensationGradient DescentNeural Networks With DelayOptimization AlgorithmsStochastic Gradient Descent
Deep neural network (DNN) learns hierarchical representations from big data in a multilayer network structure and has achieved great successes in many fields such as computer vision and speech analysis. Since DNN usually contains several billions of parameters, the asynchronous stochastic gradient descent (ASGD) algorithm is often used to train an effective DNN model on a computer cluster. However, as the increase of computing nodes and data size, ASGD suffers from serious slow convergence deficiency because the parameters might be wrongly updated by long-term delayed gradients. In this paper, we propose a delay compensated asynchronous Adam (DC-Adam) algorithm to train DNN. In particular, DC-Adam updates the parameters with the moment increment which is the division of the first and the second moments to take the advantage of the original Adam algorithm, and compensates the gradient with the first-order component in its Taylor expansion. Since the delay compensation technique reduces the error of delayed gradients, and the moment increment further counteracts the influence of approximated compensation, DC-Adam converges much more rapidly than ASGD on a computer cluster with moderate computing nodes. We theoretically analyze the Ergodic convergence rate of DC-Adam and compare with DC-ASGD. We implement our DC-Adam algorithm on 61 computing nodes in a computer cluster, and conduct image classification by using LeNet and ResNet, respectively, on the MNIST and CIFAR-10 datasets. The experimental results demonstrate that DC-Adam greatly accelerates the training progress and achieves almost linear speedup rate as increasing the computing nodes.
17
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Yangqing Jia, Evan Shelhamer, Jeff Donahue et al. · 2014 · 11.2K citations