IEEE Transactions on Parallel and Distributed Systems · 2011 · 280 citations · 23 references
ComputingCloud ComputingComputer ScienceCloud ArchitectureCloud InfrastructureParallel ComputingCloud Resource ManagementCompute Resource ManagementJob SchedulingParallel Computing SystemsResource SchedulingWorkload SchedulingCloud Computing ArchitecturesData ProcessingDynamic Resource AllocationMapreduceParallel Data ProcessingParallel ProcessingParallel Task ExecutionVirtual Machines
Parallel data processing has become a key application for IaaS clouds, yet current frameworks are built for static, homogeneous clusters and often allocate insufficient resources, inflating time and cost. This work explores the opportunities and challenges of efficient parallel data processing in clouds and introduces the Nephele research project. Nebhele is the first framework to leverage dynamic resource allocation for task scheduling and execution, assigning tasks to various VM types that are automatically instantiated and terminated, and its performance is evaluated against Hadoop on MapReduce‑inspired jobs.
In recent years ad hoc parallel data processing has emerged to be one of the killer applications for Infrastructure-as-a-Service (IaaS) clouds. Major Cloud computing companies have started to integrate frameworks for parallel data processing in their product portfolio, making it easy for customers to access these services and to deploy their programs. However, the processing frameworks which are currently used have been designed for static, homogeneous cluster setups and disregard the particular nature of a cloud. Consequently, the allocated compute resources may be inadequate for big parts of the submitted job and unnecessarily increase processing time and cost. In this paper, we discuss the opportunities and challenges for efficient parallel data processing in clouds and present our research project Nephele. Nephele is the first data processing framework to explicitly exploit the dynamic resource allocation offered by today's IaaS clouds for both, task scheduling and execution. Particular tasks of a processing job can be assigned to different types of virtual machines which are automatically instantiated and terminated during the job execution. Based on this new framework, we perform extended evaluations of MapReduce-inspired processing jobs on an IaaS cloud system and compare the results to the popular data processing framework Hadoop.
23
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Michael Isard, Mihai Budiu, Yuan Yu et al. · 2007 · 2.5K citations
Christopher Olston, Benjamin Reed, Utkarsh Srivastava et al. · 2008 · 1.7K citations
kvm : the Linux Virtual Machine Monitor
Avi Kivity · 2007 · 1.3K citations