Context-aware Scheduling for Apache Hadoop over Pervasive Environments

Abstract

This article proposes to improve Apache Hadoop scheduling through the usage of context-awareness. Apache Hadoop is the most popular implementation of the MapReduce paradigm for distributed computing, but its design doesn’t adapt automatically to computing nodes’ context and capabilities. By introducing context-awareness into Hadoop, we intent to dynamically adapt its scheduling to the execution environment. This is a necessary feature in the context of pervasive grids, which are heterogeneous, dynamic and shared environments. The solution has been incorporated into Hadoop and evaluated through controlled experi- ments. The experiments demonstrate that context-awareness provides comparative performance gains, especially when part of the resources disappear during execution.