2014 · 26 citations · 34 references
ComputingCloud ComputingHigh-performance ComputingCloud ArchitectureDistributed SystemsParallel ComputingDistributed ComputingDistributed SchedulingJob SchedulingResource SchedulingScalability ArchitectureWorkload SchedulingDistributed Job ExecutionDistributed Task SchedulingGranularityJob QueuesMessage QueuesScalabilityScalable SchedulingSystem Utilization
Task scheduling and execution over large scale, distributed systems plays an important role on achieving good performance and high system utilization. Due to the explosion of parallelism found in today's hardware, applications need to perform over-decomposition to deliver good performance, this over-decomposition is driving job management systems' requirements to support applications with a growing number of tasks with finer granularity. Our goal in this work is to provide a compact, light-weight, scalable, and distributed task execution framework (Cloud Kon) that builds upon cloud computing building blocks (Amazon EC2, SQS, and Dynamo DB). Most of today's state-of-the-art job execution systems have predominantly Master/Slaves architectures, which have inherent limitations, such as scalability issues at extreme scales and single point of failures. On the other hand distributed job management systems are complex, and employ non-trivial load balancing algorithms to maintain good utilization. Cloud Kon is a distributed job management system that can support both HPC and MTC workloads with millions of tasks/jobs. We compare our work with other state-of-the-art job management systems including Sparrow and MATRIX. The results show that Cloud Kon delivers better scalability compared to other state-of-the-art systems for some metrics - all with a significantly smaller code-base (5%).
34
Spark: cluster computing with working sets
Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin et al. · 2010 · 4.2K citations
Condor-G: A Computation Management Agent for Multi-Institutional Grids
James Frey, Todd Tannenbaum, Miron Livny et al. · Cluster Computing · 2002 · 785 citations
Sergey Melnik, Andrey Gubarev, Jing Jing Long et al. · Proceedings of the VLDB Endowment · 2010 · 709 citations