Is Network the Bottleneck of Distributed Training?

Abstract

Recently there has been a surge of research on improving the communication efficiency of distributed training. However, little work has been done to systematically understand whether the network is the bottleneck and to what extent.