BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer

TLDR

Big data from scientific instruments generates massive volumes that must be collected, indexed, archived, shared, and analyzed, yet existing bulk transfer tools fail to meet the high‑performance, deadline‑aware demands of extreme‑scale science due to disjoint transfer loops, cross‑interference, and lack of QoS awareness. The BigData Express project seeks to deliver a schedulable, predictable, and high‑performance data‑transfer service that supports a federation of big‑data science platforms. Its software is being deployed and evaluated at institutions such as UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, and Ciena, and is being integrated with the StarLight International/National Communications Exchange Facility across Pacific, National, and Global Research Platforms.

Abstract

Big Data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed, highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. Although significant improvements have been made in the area of bulk data transfer, the currently available data transfer tools and services can not successfully address the high-performance and time-constraint challenges of data transfer required by extreme-scale science applications for the following reasons: disjoint end-to-end data transfer loops, cross-interference between data transfers, and existing data transfer tools and services are oblivious to user requirements (deadline and QoS requirements). Fermilab has been working on the BigData Express project to address these problems. BigData Express seeks to provide a schedulable, predictable, and high-performance data transfer service for big data science. The BigData Express software is being deployed and evaluated at multiple research institutions, which include UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, Ciena, and other sites. Meanwhile, the BigData Express research team is collaborating with the StarLight International/National Communications Exchange Facility to deploy BigData Express at various research platforms, including Pacific Research Platform, National Research Platform, and Global Research Platform. It is envisioned that we are working toward building a high-performance data transfer federation for big data science.