The Ring Buffer Network Bus (RBNB) DataTurbine Streaming Data Middleware for Environmental Observing Systems

Abstract

The environmental science and engineering communities are actively engaged in planning and developing the next generation of large-scale sensor-based observing systems. These systems face two significant challenges: heterogeneity of instrumentation and complexity of data stream processing. Environmental observing systems incorporate instruments across the spectrum of complexity, from temperature sensors to acoustic Doppler current profilers, to streaming video cameras. Managing these instruments and their data streams is a serious challenge. Critical infrastructure requirements common to all of these sensor-based observing systems are reliable data transport, the promotion of sensors and sensor streams to first-class objects, a framework for the integration of heterogeneous instruments, and a comprehensive suite of services for data management, routing, synchronization, monitoring, and visualization. In this paper we present the RBNB DataTurbine, an open-source streaming data middleware system, and discuss how the RBNB DataTurbine satisfies the critical cyberinfrastructure requirements core to these sensor-based observing systems. The discussion includes the results from real-world deployments.