Proceedings of the VLDB Endowment · 2014 · 78 citations · 37 references
ComputingComputer ScienceComputer SystemsDistributed SystemsParallel AlgorithmsParallel ComputingData ParallelismData PartitioningDistributed ComputingQuery ProcessingCompetitive RatioData DistributionLocal StatePartitioningPartitioning SchemesPartitioning StrategiesShared-nothing ArchitectureStatic Partitioning
Scalable join processing in a parallel shared-nothing environment requires a partitioning policy that evenly distributes the processing load while minimizing the size of state maintained and number of messages communicated. Previous research proposes static partitioning schemes that require statistics beforehand. In an online or streaming environment in which no statistics about the workload are known, traditional static approaches perform poorly. This paper presents a novel parallel online dataflow join operator that supports arbitrary join predicates. The proposed operator continuously adjusts itself to the data dynamics through adaptive dataflow routing and state repartitioning. The operator is resilient to data skew, maintains high throughput rates, avoids blocking behavior during state repartitioning, takes an eventual consistency approach for maintaining its local state, and behaves strongly consistently as a black-box dataflow operator. We prove that the operator ensures a constant competitive ratio 3:75 in data distribution optimality and that the cost of processing an input tuple is amortized constant, taking into account adaptivity costs. Our evaluation demonstrates that our operator outperforms the state-of-the-art static partitioning schemes in resource utilization, throughput, and execution time.
37
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
The Design of the Borealis Stream Processing Engine
Daniel J. Abadi, Yanif Ahmad, Magdalena Bałazińska et al. · 2005 · 1.2K citations
A comparison of approaches to large-scale data analysis
Andrew Pavlo, Erik K. Paulson, Alexander Rasin et al. · 2009 · 1.1K citations
Joseph M. Hellerstein, Peter J. Haas, Helen J. Wang · 1997 · 928 citations · Full text
Ron Avnur, Joseph M. Hellerstein · ACM SIGMOD Record · 2000 · 795 citations