2013 · 70 citations · 42 references
ComputingComputer ScienceComputer SystemsCluster ComputingDistributed SystemsSchedulingCluster ManagementCompute Resource ManagementDistributed ComputingJob SchedulingResource SchedulingWorkload SchedulingCluster SchedulingDeadlinesMapreduceMapreduce AlgorithmsPreemptionPreemption PoliciesRuntime SchedulingWork-conserving Scheduling
This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7%, and is preferable to existing approaches.
42
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Christopher Olston, Benjamin Reed, Utkarsh Srivastava et al. · 2008 · 1.7K citations
Improving MapReduce performance in heterogeneous environments
Matei Zaharia, Andy Konwinski, Anthony D. Joseph et al. · UC Berkeley · 2008 · 1.6K citations
Mesos: a platform for fine-grained resource sharing in the data center
Benjamin Hindman, Andy Konwinski, Matei Zaharia et al. · UC Berkeley · 2011 · 1.6K citations