MPI Collectives and Datatypes for Hierarchical All-to-all Communication

Abstract

With recent MPI 3.0 functionality for creating communicators that partly reflects the hierarchy of standard clusters of shared-memory nodes, hierarchical, collective algorithms can more conveniently be implemented by combinations of other collective MPI operations. On systems that support MPI 3.0, with MPI_Alltoall as a concrete example, we show that performance on par with or better than native MPI implementations is possible, thus illustrating that the provided hierarchy awareness can be an effective and portable means for applications to implement their own, efficient (non-MPI) collective operations. Parts of the MPI_Alltoall implementation relies on MPI derived datatypes; however, the MPI collective interfaces lack expressivity to take full advantage of the performance benefits offered by the derived datatype mechanism.