Evaluation and Performance Modeling of a Burst Buffer Solution

Abstract

Hierarchical storage architectures are required to meet both, capacity and bandwidth requirements for future high-end storage architectures. In this paper we present the results of an evaluation of an emerging technology, DataDirect Networks' (DDN) Infinite Memory Engine (IME). IME allows to realize a fast buffer in front of a large capacity storage system. We collected benchmarking data with IOR and with the HPC application NEST. The IOR bandwidth results show how well network bandwidth towards such fast buffer can be exploited compared to the external storage system. The NEST benchmarks clearly demonstrate that IME can reduce I/O-induced load imbalance between MPI ranks to a minimum while speeding up I/O as a whole by a considerable factor. In addition to these direct measurements, a performance model for NEST is developed. In combination with a generic and abstract burst buffer architecture, this model generates predictions about appropriate burst buffer and I/O parameters to achieve specific performance goals for NEST on HPC clusters of varying size. Specifically, it is investigated in which parameter range burst buffers are able to counteract the widening performance gap between compute and I/O.