Towards optimizing large-scale data transfers with end-to-end integrity verification

TLDR

The scale of scientific data is rapidly growing, requiring fast, reliable transfers to remote facilities, but end‑to‑end integrity checks add overhead that lengthens transfer time. This paper evaluates strategies to maximize overlap between data transfer and checksum computation. We assess file‑level and block‑level pipelining of GridFTP transfers, using theoretical analysis and real experiments to compare overlap and performance. Block‑level pipelining can reduce overall transfer time with integrity verification by up to 70 % versus sequential execution, and by up to 60 % versus file‑level pipelining.

Abstract

The scale of scientific data generated by experimental facilities and simulations on high-performance computing facilities has been growing rapidly. In many cases, this data needs to be transferred rapidly and reliably to remote facilities for storage, analysis, sharing etc. At the same time, users want to verify the integrity of the data by doing a checksum after the data has been written to disk at the destination, to ensure the file has not been corrupted, for example due to network or storage data corruption, software bugs or human error. This end-to-end integrity verification creates additional overhead (extra disk I/O and more computation) and increases the overall data transfer time. In this paper, we evaluate strategies to maximize the overlap between data transfer and checksum computation. More specifically, we evaluate file-level and block-level (with various block sizes) pipelining to overlap data transfer and checksum computation. We evaluate these pipelining approaches in the context of GridFTP, a widely used protocol for science data transfers. We conducted both theoretical analysis and real experiments to evaluate our methods. The results show that block-level pipelining is an effective method in maximizing the overlap between data transfer and checksum computation and can improve the overall data transfer time with end-to-end integrity verification by up to 70% compared to the sequential execution of transfer and checksum, and by up to 60% compared to file-level pipelining.