Workflow Skeletons: Increasing Scalability of Scientific Workflows by Combining Orchestration and Choreography

Abstract

Dataflow modeling is the natural way of composing scientific workflows, because they often comprise numerous data transformation steps applying massive parallelism. However, modeling control flow within dataflow is often achieved at the expense of clarity and comprehensibility. This paper describes scientific workflows maintaining the robustness of centralized control (using orchestration) by modeling control flow, while at the same time integrating sub-workflows that are modeled by workflow skeletons (using choreography) describing dataflow. Following the concept of algorithmic skeletons, we define workflow skeletons as re-usable parallel constructs describing dataflow connections between proxies representing services. Proxies are able to communicate with each other allowing for efficient coupling between parallel tasks and avoiding of unnecessary data transfers. Skeletons increase scalability on demand by accepting the number of parallel tasks. The primary contributions are a formal model describing workflow skeletons and a script language "Work flow Skeleton Language" (WorkSKEL). Furthermore, this paper demonstrates the definition of selected workflow patterns like pipeline and farm in WorkSKEL.