2012 · 26 citations · 10 references
ComputingComputer ScienceComputer SystemsData ManagementDistributed SystemsParallel ComputingDistributed Data ProcessingHigh-throughput ComputingJob SchedulingLatency OptimizationThroughput OptimizationWorkload SchedulingCleanupData-intensive ComputingEvent NotificationsLarge-scale Data ProcessingMapreduceParallel Performance OptimizationParallel Task ExecutionTask Assignment
Hadoop MapReduce is a widely used parallel computing framework for solving data-intensive problems. To be able to process large-scale datasets, the fundamental design of the standard Hadoop places more emphasis on high-throughput of data than on job execution performance. This causes performance limitation when we use Hadoop MapReduce to execute short jobs that requires quick responses. In order to speed up the execution of short jobs, this paper proposes optimization methods to improve the execution performance of MapReduce jobs. We made three major optimizations: first, we reduce the time cost during the initialization and termination stages of a job by optimizing its setup and cleanup tasks, second, we replace the pull-model task assignment mechanism with a push-model, third, we replace the heartbeat-based communication mechanism with an instant message communication mechanism for event notifications between the Job Tracker and Task Trackers. Experimental results show that the job execution performance of our improved version of Hadoop is about 23% faster on average than the standard Hadoop for our test application.
10
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Fine particles, thin films and exchange anisotropy
I. S. Jacobs · Magnetism · 1963 · 2.8K citations · Full text
Improving MapReduce performance in heterogeneous environments
Matei Zaharia, Andy Konwinski, Anthony D. Joseph et al. · UC Berkeley · 2008 · 1.6K citations
Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma et al. · 2010 · 1.4K citations
Interpreting the Data: Parallel Analysis with Sawzall
Rob Pike, Sean Dorward, Robert Griesemer et al. · Scientific Programming · 2005 · 616 citations · Full text