Proceedings of the 48th International Conference on Parallel Processing

Abstract

The critical path of internode communication on large-scale systems is\ncomposed of multiple components. When a supercomputing application initiates\nthe transfer of a message using a high-level communication routine such as an\nMPI_Send, the payload of the message traverses multiple software stacks, the\nI/O subsystem on both the host and target nodes, and network components such as\nthe switch. In this paper, we analyze where, why, and how much time is spent on\nthe critical path of communication by modeling the overall injection overhead\nand end-to-end latency of a system. We focus our analysis on the performance of\nsmall messages since fine-grained communication is becoming increasingly\nimportant with the growing trend of an increasing number of cores per node. The\nanalytical models present an accurate and detailed breakdown of time spent in\ninternode communication. We validate the models on Arm ThunderX2-based servers\nconnected with Mellanox InfiniBand. This is the first work of this kind on Arm.\nAlongside our breakdown, we describe the methodology to measure the time spent\nin each component so that readers with access to precise CPU timers and a PCIe\nanalyzer can measure breakdowns on systems of their interest. Such a breakdown\nis crucial for software developers, system architects, and researchers to guide\ntheir optimization efforts. As researchers ourselves, we use the breakdown to\nsimulate the impacts and discuss the likelihoods of a set of optimizations that\ntarget the bottlenecks in today's high-performance communication.\n