2017 · 33 citations · 13 references
ComputingComputer ScienceComputer SystemsCluster ComputingDistributed SystemsParallel ComputingData ParallelismDistributed ComputingDistributed Data ProcessingLatency OptimizationReal-time Resource ManagementStream ProcessingBandwidthDistributed Stream ProcessingElastic Resource ScalingElastic ScalingElastic Stream ProcessingJitterLatency JitterParallel Processing
High-volume data streams are straining the limits of stream processing frameworks which need advanced parallel processing capabilities to withstand the actual incoming bandwidth. Parallel processing must be synergically integrated with elastic features in order dynamically scale the amount of utilized resources by accomplishing the Quality of Service goals in a cost-effective manner. This paper proposes a control-theoretic strategy to drive the elastic behavior of latency-sensitive streaming operators in distributed environments. The strategy takes scaling decisions in advance by relying on a predictive model-based approach. Our ideas have been experimentally evaluated on a cluster using a real-world streaming application fed by synthetic and real datasets. The results show that our approach takes the strictly necessary reconfigurations while providing reduced resource consumption. Furthermore, it allows the operator to meet desired average latency requirements with a significant reduction in the experienced latency jitter.
13
Distributed caching with memcached
Brad Fitzpatrick · Linux journal · 2004 · 513 citations
Flux: an adaptive partitioning operator for continuous query systems
Mehul A. Shah, Joseph M. Hellerstein, Sirish Chandrasekaran et al. · 2004 · 317 citations
Elastic Scaling for Data Stream Processing
Buğra Gedik, Scott Schneider, Martin Hirzel et al. · IEEE Transactions on Parallel and Distributed Systems · 2014 · 236 citations