2011 · 16 citations · 18 references
ComputingComputer ScienceInformation TechnologyDistributed SystemsGrid ComputingParallel ComputingData OrchestrationDistributed ComputingDistributed Data ProcessingJob SchedulingParallel Runtime SystemsWorkflow OrchestrationApi DesignData-intensive ComputingMapreduceRuntime SupportRuntime System ArchitectureSatellite Climate Data AnalysisScheduler DesignScientific Workflow Management
MapReduce is promising for developing both scalable business and scientific data intensive applications. However, there are few existing scientific workflow systems which can benefit from the MapReduce programming model. We propose a workflow system for integrating structure, and orchestrating MapReduce jobs for scientific data intensive workflows. The system consists of a simple workflow design C++ API, a job scheduler, and a runtime support system for Hadoop or Sector/Sphere frameworks. A climate satellite data intensive processing and analysis application is developed as a use case and an evaluation for the workflow system. The evaluation shows that it is possible to make the steps in the climate data intensive application automatically from data gridding to complex data analysis using the workflow system. The performance of the climate analysis application is significantly improved by the enabled MapReduce workflow system compared with the sequential embarrassing parallel methods. The overhead of the workflow system is negligible. However, the graphic user interface is still under development for the workflow system.
18
Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Ewa Deelman, Gurmeet Singh, Mei-Hui Su et al. · Scientific Programming · 2005 · 1.2K citations · Full text
CloudBurst: highly sensitive read mapping with MapReduce
Michael C. Schatz · Bioinformatics · 2009 · 631 citations · Full text
Scientific workflow applications on Amazon EC2
Gideon Juve, Ewa Deelman, Karan Vahi et al. · 2009 · 254 citations · Full text