Bridging high velocity and high volume industrial big data through distributed in-memory storage & analytics

Abstract

With an exponential increase in time series sensor data generated by an ever-growing number of sensors on industrial equipment, new systems are required to efficiently store and analyze this “Industrial Big Data.” To actively monitor industrial equipment there is a need to process large streams of high velocity time series sensor data as it arrives, and then store that data for subsequent analysis. Historically, separate systems would meet these needs, with neither system having the ability to perform fast analytics incorporating both just-arrived and historical data. In-memory data grids are a promising technology that can support both near real-time analysis and mid-term storage of big datasets, bridging the gap between high velocity and high volume big time series sensor data. This paper describes the development of a prototype infrastructure with an in-memory data grid at its core to analyze high velocity (>100,000 points per second), high volume (TB's) time series data produced by a fleet of gas turbines monitored at GE Power & Water's Remote Monitoring & Diagnostics Center.