Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)

TLDR

Scientific code performance varies with runtime platform, configuration, IO API, and file system. The paper introduces ADIOS to let developers select optimal IO routines without recompilation, reducing time and resource costs. ADIOS uses a library of tuned IO routines and transport methods configured via an XML file at startup, enabling dynamic selection among MPI‑IO, POSIX, asynchronous, and in situ visualization options, and has been adapted for seven major scientific codes. ADIOS achieves high performance and scalability, enabling seamless integration with workflow and visualization systems, and achieved 20 GB/s write throughput on the Jaguar Cray XT4—about half peak performance.

Abstract

Scientific codes are all subject to variation in performance depending on the runtime platform and/or configuration, the output writing API employed, and the file system for output. Since changing the IO routines to match the optimal or desired configuration for a given system can be costly in terms of human time and machine resources, the Adaptable IO System provides an API nearly as simple as POSIX IO that also provides developers with the flexibility of selection the optimal IO routines for a given platform, without recompilation. As a side effect, we also gain the ability to transparently integrate more tightly with workflow systems like Kepler and Pegasus and visualization systems like Visit with no runtime impact. We achieve this through our library of highly tuned IO routines and other transport methods selected and configured in an XML file read only at startup. ADIOS-based IO has demonstrated high levels of performance and scalability. For example, we have achieved 20 GB/sec write performance using GTC on the Jaguar Cray XT4 system at Oak Ridge National Labs (about 50\% of peak performance). We can change GTC output among MPI-IO synchronous, MPI-IO collective, POSIX IO, no IO (for baseline testing), asynchronous IO using the Georgia Tech DataTap system, and Visit directly for in situ visualization with no changes to the source code. We designed this initial version of ADIOS based on the data requirements of 7 major scientific codes (GTC, Chimera, GTS, XGC1, XGC0, FLASH, and S3D) and have successfully adapted all of them to use ADIOS for all of their IO needs.