IEEE Transactions on Parallel and Distributed Systems · 2012 · 321 citations · 22 references
ComputingComputer ScienceComputer SystemsDistributed SystemsParallel AlgorithmsParallel ComputingDistributed ComputingDistributed Data ProcessingQuery ProcessingScalability ArchitectureStream ProcessingStreaming AlgorithmsData Stream ProcessingElastic ScalabilityElastic Stream ProcessingLoadParallelizationStream Processing ScalabilityStream Query ProcessingStreaming Data Processing
Many applications in telecommunications, network security, and large‑scale sensor networks require online processing of continuous data flows that generate high loads, yet current stream processing engines fail to scale due to single‑node bottlenecks and static configurations that cause under‑ or over‑provisioning. This paper introduces StreamCloud, a scalable and elastic stream processing engine designed to handle large data stream volumes. StreamCloud employs a novel parallelization technique that splits queries into subqueries assigned to independent node sets to reduce distribution overhead, uses low‑intrusiveness elastic protocols to adjust resources to incoming load, and combines elasticity with dynamic load balancing to minimize computational resources. The authors present the system design, implementation, and a comprehensive evaluation demonstrating StreamCloud’s scalability and elasticity.
Many applications in several domains such as telecommunications, network security, large-scale sensor networks, require online processing of continuous data flows. They produce very high loads that requires aggregating the processing capacity of many nodes. Current Stream Processing Engines do not scale with the input load due to single-node bottlenecks. Additionally, they are based on static configurations that lead to either under or overprovisioning. In this paper, we present StreamCloud, a scalable and elastic stream processing engine for processing large data stream volumes. StreamCloud uses a novel parallelization technique that splits queries into subqueries that are allocated to independent sets of nodes in a way that minimizes the distribution overhead. Its elastic protocols exhibit low intrusiveness, enabling effective adjustment of resources to the incoming load. Elasticity is combined with dynamic load balancing to minimize the computational resources used. The paper presents the system design, implementation, and a thorough evaluation of the scalability and elasticity of the fully implemented system.
22
Aurora: a new model and architecture for data stream management
Daniel J. Abadi, Don Carney, Ugur �etintemel et al. · The VLDB Journal · 2003 · 1.5K citations
The Design of the Borealis Stream Processing Engine
Daniel J. Abadi, Yanif Ahmad, Magdalena Bałazińska et al. · 2005 · 1.2K citations
S4: Distributed Stream Computing Platform
Leonardo Neumeyer, Bruce Robbins, Anish Nair et al. · 2010 · 905 citations
Sirish Chandrasekaran, Owen Cooper, Amol Deshpande et al. · 2003 · 735 citations