Latency modeling and minimization for large-scale scientific workflows in distributed network environments

Abstract

Large-scale e-science applications feature complex workflows consisting of many computing modules. Mapping such workflows in distributed network environments and minimizing their latency are crucial to those applications that require fast system response and prompt user interaction. We model the time cost of each workflow component and design an efficient algorithm to compute the exact end-to-end delay of the entire workflow by explicitly accounting for the resource sharing dynamics. We further propose a workflow mapping approach to minimize the workflow latency using a recursive optimization procedure. The validity of the cost models and the accuracy of the latency computing algorithm are verified in comparison with an approximate solution, a dynamic system simulation program, and a workflow engine deployed in a real network. The performance superiority of the proposed mapping approach is illustrated by extensive simulation-based comparisons with existing algorithms.