Improving Load Balance for Data-Intensive Computing on Cloud Platforms

Abstract

Nowadays, big data problems are ubiquitous, which in turn creates huge demand for data-intensive computing. The advent of Cloud Computing has made data-intensive computing much more accessible and affordable than ever before. One of the crucial issues that can significantly affect the performance of data-intensive applications is the load balance among cluster nodes. In this paper, we address the load balance problem in the context of Hadoop Distributed File System (HDFS), a widely used file system for data-intensive computing on Cloud platforms, and present an innovative replica placement policy for HDFS, which can perfectly balance the computing load among all cluster nodes in both homogeneous and heterogeneous cluster environments.