Chiron: a parallel engine for algebraic scientific workflows

TLDR

Large‑scale scientific experiments rely on scientific workflows managed by SWfMS, but few support parallel execution, and those that do are labor‑intensive with limited optimization primitives. The authors developed workflow algebra to enable optimized parallel execution of scientific workflows. Chiron implements the workflow algebra with a native distributed provenance system and was evaluated in two studies comparing performance and scalability against other approaches. The studies show that Chiron efficiently executes scientific workflows, offering declarative specification and runtime provenance support. © 2013 John Wiley & Sons, Ltd.

Abstract

SUMMARY Large‐scale scientific experiments based on computer simulations are typically modeled as scientific workflows, which eases the chaining of different programs. These scientific workflows are defined, executed, and monitored by scientific workflow management systems (SWfMS). As these experiments manage large amounts of data, it becomes critical to execute them in high‐performance computing environments, such as clusters, grids, and clouds. However, few SWfMS provide parallel support. The ones that do so are usually labor‐intensive for workflow developers and have limited primitives to optimize workflow execution. To address these issues, we developed workflow algebra to specify and enable the optimization of parallel execution of scientific workflows. In this paper, we show how the workflow algebra is efficiently implemented in Chiron, an algebraic based parallel scientific workflow engine. Chiron has a unique native distributed provenance mechanism that enables runtime queries in a relational database. We developed two studies to evaluate the performance of our algebraic approach implemented in Chiron; the first study compares Chiron with different approaches, whereas the second one evaluates the scalability of Chiron. By analyzing the results, we conclude that Chiron is efficient in executing scientific workflows, with the benefits of declarative specification and runtime provenance support. Copyright © 2013 John Wiley & Sons, Ltd.