Application-Aware Network Design for Hadoop MapReduce Optimization Using Software-Defined Networking

Abstract

The MapReduce (M/R) framework used in Hadoop has become the de facto standard for big data analytics. However, the lack of network-awareness of the default M/R resource manager in a traditional IP network can cause unbalanced job scheduling and network bottlenecks; such factors can eventually lead to an increase in the Hadoop M/R job completion time. In this paper, we propose a software-defined network (SDN) approach in an application-aware network (AAN) platform that provides both underlying networks functions as well M/R particular forwarding logics. We measure the resources' usage for M/R workloads using the HiBench benchmark suite to identify the traffic pattern. We then demonstrate that by using our AAN-SDN framework, which uses an adaptive traffic engineering mechanism, the job completion time can be noticeably improved.