Using InfiniBand Hardware Gather-Scatter Capabilities to Optimize MPI All-to-All

Abstract

The MPI all-to-all algorithm is a data intensive, high-cost collective algorithm used by many scientific High Performance Computing applications. Optimizations for small data exchange use aggregation techniques, such as the Bruck algorithm, to minimize the number of messages sent, and minimize overall operation latency. This paper presents three variants of the Bruck algorithm, which differ in the way data is laid out in memory at intermediate steps of the algorithm. Mellanox's InfiniBand support for Host Channel Adapter (HCA) hardware scatter/gather is used selectively to replace CPU-based buffer packing and unpacking. Using this offload capability reduces the eight and sixteen byte all-to-all latency on 1024 MPI Processes by 9.7% and 9.1%, respectively. The optimization accounts for a decrease in the total memory handling time of 40.6% and 57.9%, respectively.