Error Compensated Quantized SGD and its Applications to Large-scale\n Distributed Optimization

Abstract

Large-scale distributed optimization is of great importance in various\napplications. For data-parallel based distributed learning, the inter-node\ngradient communication often becomes the performance bottleneck. In this paper,\nwe propose the error compensated quantized stochastic gradient descent\nalgorithm to improve the training efficiency. Local gradients are quantized to\nreduce the communication overhead, and accumulated quantization error is\nutilized to speed up the convergence. Furthermore, we present theoretical\nanalysis on the convergence behaviour, and demonstrate its advantage over\ncompetitors. Extensive experiments indicate that our algorithm can compress\ngradients by a factor of up to two magnitudes without performance degradation.\n