2016 · 18 citations · 17 references
ComputingComputer EngineeringComputer ScienceDistributed SystemsFault-tolerant ComputingHeterogeneous ComputingDistributed ComputingDistributed Data ProcessingHeterogeneous Cluster ComputingJob SchedulingLatency OptimizationSpeculative ExecutionHeterogeneous Cluster ArchitectureLarge-scale Data ProcessingMapreduceRedundant ComputationSpeculationSpeculative OptimizationSpeculative Task ExecutionStraggler Mitigation
In this paper, we aim to develop an efficient speculation framework for a heterogeneous cluster. Speculation is a common mechanism that identifies ‘slow’ node in a cluster and starts redundant tasks on other nodes to guarantee the reliability. We consider MapReduce/Hadoop as a representative computing platform, and our general goal is to accurately and quickly identify the straggler nodes during the job execution. On the one hand, our approach significantly reduces unnecessary speculative executions that occupy system resources, but do not get finished. On the other hand, when a node is prone to failure, our solution is able to detect it at an early stage and effectively launch a speculative task to avoid the delay in the job execution. We implement our solution in Hadoop platform and evaluate it with extensive experiments. The results show that our solution is efficient and effective when handling the speculative execution. The job execution time in our system is superior to that in the current Hadoop distribution.
17
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Yucheng Low, Danny Bickson, Joseph E. Gonzalez et al. · Proceedings of the VLDB Endowment · 2012 · 1.7K citations
Improving MapReduce performance in heterogeneous environments
Matei Zaharia, Andy Konwinski, Anthony D. Joseph et al. · UC Berkeley · 2008 · 1.6K citations
Kay Ousterhout, Patrick Wendell, Matei Zaharia et al. · 2013 · 582 citations · Full text
Improving MapReduce performance through data placement in heterogeneous Hadoop clusters
Jiong Xie, Shu Yin, Xiaojun Ruan et al. · 2010 · 381 citations