2018 · 26 citations · 12 references
ComputingComputer ScienceHigh-performance ComputingParallel ComputingSchedulingStorage SystemsPerformance ModelingPerformance TuningWorkflow AnalysisWorkflow DesignWorkflow ManagementWorkflow OrchestrationEfficiencyI/o EfficiencyMonitoring SystemsReal-world ApplicationWorkflow Automation FrameworksWorkflow ExecutionWorkflow Performance ManagementWorkflow Scheduling
Scientific discovery increasingly depends on complex workflows consisting of multiple phases and sometimes millions of parallelizable tasks or pipelines. These workflows access storage resources for a variety of purposes, including preprocessing, simulation output, and postprocessing steps. Unfortunately, most workflow models focus on the scheduling and allocation of computational resources for tasks while the impact on storage systems remains a secondary objective and an open research question. I/O performance is not usually accounted for in workflow telemetry reported to users. In this paper, we present an approach to augment the I/O efficiency of the individual tasks of workflows by combining workflow description frameworks with system I/O telemetry data. A conceptual architecture and a prototype implementation for HPC data center deployments are introduced. We also identify and discuss challenges that will need to be addressed by workflow management and monitoring systems for HPC in the future. We demonstrate how real-world applications and workflows could benefit from the approach, and we show how the approach helps communicate performance-tuning guidance to users.
12
Pegasus, a workflow management system for science automation
Ewa Deelman, Karan Vahi, Gideon Juve et al. · Future Generation Computer Systems · 2014 · 818 citations · Full text
Swift: A language for distributed parallel scripting
Michael Wilde, Mihael Hategan, Justin M. Wozniak et al. · Parallel Computing · 2011 · 404 citations
The future of scientific workflows
Ewa Deelman, Tom Peterka, İlkay Altıntaş et al. · The International Journal of High Performance Computing Applications · 2017 · 179 citations · Full text
Compiler Techniques for Massively Scalable Implicit Task Parallelism
Timothy G. Armstrong, Justin M. Wozniak, Michael Wilde et al. · 2014 · 54 citations