Auto-scaling techniques for elastic data stream processing

Abstract

Typical use cases like financial trading or monitoring of manufacturing equipment pose huge challenges regarding end to end latency as well as throughput towards existing data stream processing systems. Established solutions like Apache S4 or Storm need to scale out to a large set of hosts to meet these challenges. An ideal system can react to workload changes by on demand acquisition or release of hosts. Thereby, it can handle unexpected peak loads as well as improve the average utilization of the system. This property is called elasticity.