Exploring Hidden Dimensions in Parallelizing Convolutional Neural\n Networks

Abstract

The past few years have witnessed growth in the computational requirements\nfor training deep convolutional neural networks. Current approaches parallelize\ntraining onto multiple devices by applying a single parallelization strategy\n(e.g., data or model parallelism) to all layers in a network. Although easy to\nreason about, these approaches result in suboptimal runtime performance in\nlarge-scale distributed training, since different layers in a network may\nprefer different parallelization strategies. In this paper, we propose\nlayer-wise parallelism that allows each layer in a network to use an individual\nparallelization strategy. We jointly optimize how each layer is parallelized by\nsolving a graph search problem. Our evaluation shows that layer-wise\nparallelism outperforms state-of-the-art approaches by increasing training\nthroughput, reducing communication costs, achieving better scalability to\nmultiple GPUs, while maintaining original network accuracy.\n