Data-Aware Resource Scheduling for Multicloud Workflows: A Fine-Grained Simulation Approach

Abstract

Cloud infrastructures have seen increasing popularity for addressing the growing computational needs of today's scientific and engineering applications. However, resource management challenges exist in the elastic cloud environment, such as resource provisioning and task allocation, especially when data movement between multiple domains plays an important role. In this work, we study the impact of data-aware resource management and scheduling on scientific workflows in multicloud environments. We develop a workflow simulator based on a network simulation framework for fine-grained simulation for workflow computation and data movement. Using the workload traces from a production metagenomic data analysis service, we evaluate different resource scheduling mechanisms, including proposed data-aware scheduling policies under various resource and bandwidth configurations. The results of this work are expected to answer questions about how to provision computing resources for certain workloads efficiently and how to place tasks across multidomain clouds in order to reduce data movement costs for overall improved system performance.