A comparative experimental study of parallel file systems for large-scale data processing

Abstract

Large-scale scientic and business applications require data processing of ever-increasing amounts of data, fu-eling a demand for scalable parallel le systems com-prising hundreds to thousands of disks. Modern parallel le system architectures however, span a large and com-plex design space. As a result, IT architects are faced with a challenge when deciding on the most appropri-ate parallel le system for a specic scientic or indus-trial application in a large-scale computing installation. Typically, the right choice depends on the characteris-tics of the application as well as the design assumptions built into a parallel le system. In this study, we take a close look at two prominent modern parallel le systems, PVFS2 and Lustre, and compare them experimentally on a range of benchmark-driven scenarios modeling specic real-world applications. 1