Procedia Computer Science · 2013 · 24 citations · 6 references
ComputingComputer EngineeringComputer ScienceCluster ComputingDistributed SystemsParallel ComputingCluster ManagementCompute Resource ManagementDistributed Data ProcessingMemory Hierarchy OptimizationParallel Runtime SystemsWorkload SchedulingData-intensive ComputingExecution Time AnalysisExecution Time EstimationMapreducePrefetchingPrefetching MechanismsPrefetching StrategiesThroughput
In this paper, we import a prefetching mechanism into MapReduce model while retaining compatibility with the native Hadoop. Given a data-intensive application running on a Hadoop MapReduce cluster, our approach estimates the execution time of each task and adaptively preloads an amount of data to the memory before the new task is assigned to the computing node. We implement a predictive schedule and prefetching (PSP) mechanism, which is integrated into the native MapReduce runtime system. We also evaluate performance on a 10-node cluster using two popular benchmarks–grep and wordcount. The PSP mechanism reduces the execution time of grep and wordcount up to 28% with an average of 19%. Moreover, the PSP model increases the overall throughput and improves the I/O utilization. Because of the limitation of length, we did not present the experiment result detail in this paper.
6
Improving MapReduce performance in heterogeneous environments
Matei Zaharia, Andy Konwinski, Anthony D. Joseph et al. · UC Berkeley · 2008 · 1.6K citations
Informed prefetching and caching
Russel H. Patterson, Garth A. Gibson, E. Ginting et al. · 1995 · 647 citations
Improving MapReduce performance through data placement in heterogeneous Hadoop clusters
Jiong Xie, Shu Yin, Xiaojun Ruan et al. · 2010 · 381 citations
HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment
Sangwon Seo, Ingook Jang, Kyungchang Woo et al. · 2009 · 127 citations