The Globus Striped GridFTP Framework and Server

TLDR

The GridFTP extensions to the File Transfer Protocol define a general‑purpose mechanism for secure, reliable, high‑performance data movement. The authors present the Globus striped GridFTP framework, a set of client and server libraries intended to support the construction of data‑intensive tools and applications. The framework and its striped GridFTP server are designed with modular client and server libraries that enable high‑performance, secure data transfer. The server outperforms other FTP servers, achieving up to 27.3 Gbit/s memory‑to‑memory and 17 Gbit/s disk‑to‑disk over a 60 ms round‑trip, supports 1800 concurrent clients, and its performance and modularity make it a strong foundation for building tools and a testbed for innovative data‑management techniques.

Abstract

The GridFTP extensions to the File Transfer Protocol define a general-purpose mechanism for secure, reliable, high-performance data movement. We report here on the Globus striped GridFTP framework, a set of client and server libraries designed to support the construction of data-intensive tools and applications. We describe the design of both this framework and a striped GridFTP server constructed within the framework. We show that this server is faster than other FTP servers in both single-process and striped configurations, achieving, for example, speeds of 27.3 Gbit/s memory-to-memory and 17 Gbit/s disk-to-disk over a 60 millisecond round trip time, 30 Gbit/s network. In another experiment, we show that the server can support 1800 concurrent clients without excessive load. We argue that this combination of performance and modular structure make the Globus GridFTP framework both a good foundation on which to build tools and applications, and a unique testbed for the study of innovative data management techniques and network protocols.