A linear speedup analysis of distributed deep learning with sparse and quantized communication

TLDR

Distributed SGD for deep neural networks is bottlenecked by large communication overhead, and although sparsification and quantization have been shown to reduce communication cost, their impact on convergence rate remains unclear. This study investigates the convergence rate of distributed SGD for non‑convex optimization using sparse parameter averaging and gradient quantization, and proposes the periodic quantized averaging (PQASGD) strategy to further reduce communication cost while preserving convergence. The authors analyze distributed SGD with sparse averaging and quantization, and introduce PQASGD, a periodic quantized averaging scheme that maintains the O(1/√MK) convergence rate. They demonstrate that an O(1/.

Abstract

The large communication overhead has imposed a bottleneck on the performance of distributed Stochastic Gradient Descent (SGD) for training deep neural networks. Previous works have demonstrated the potential of using gradient sparsification and quantization to reduce the communication cost. However, there is still a lack of understanding about how sparse and quantized communication affects the convergence rate of the training algorithm. In this paper, we study the convergence rate of distributed SGD for non-convex optimization with two communication reducing strategies: sparse parameter averaging and gradient quantization. We show that O(1/√MK) convergence rate can be achieved if the sparsification and quantization hyperparameters are configured properly. We also propose a strategy called periodic quantized averaging (PQASGD) that further reduces the communication cost while preserving the O(1/√MK) convergence rate. Our evaluation validates our theoretical results and shows that our PQASGD can converge as fast as full-communication SGD with only 3% - 5% communication data size.