2014 · 58 citations · 19 references
ComputingComputer ScienceHigh-performance ComputingDistributed SystemsParallel AlgorithmsParallel ComputingCompute Resource ManagementDistributed ComputingDistributed SchedulingJob SchedulingTask SchedulingWorkload SchedulingDistributed Task SchedulingHigh-performance Computing SystemsKey-value StoresParallel Job SchedulingScalable Resource ManagementScalable SchedulingThroughputWorkload Management
With the exponential growth of supercomputers in parallelism, applications are growing more diverse, including traditional large-scale HPC MPI jobs, and ensemble workloads such as finer-grained many-task computing (MTC) applications. Delivering high throughput and low latency for both workloads requires developing a distributed job management system that is magnitudes more scalable than today's centralized ones. In this paper, we present a distributed job launch prototype, SLURM++, which is comprised of multiple controllers with each one managing a partition of SLURM daemons, while ZHT (a distributed key-value store) is used to store the job and resource metadata. We compared SLURM++ with SLURM using micro-benchmarks of different job sizes up to 500 nodes, with excellent results showing 10X higher throughput. We also studied the potential of distributed scheduling through simulations up to millions of nodes.
19
Kay Ousterhout, Patrick Wendell, Matei Zaharia et al. · 2013 · 582 citations · Full text
Sun Grid Engine: towards creating a compute power grid
Wolfgang Gentzsch · 2002 · 378 citations
Swift: Fast, Reliable, Loosely Coupled Parallel Computation
Yong Zhao, Mihael Hategan, Ben Clifford et al. · 2007 · 355 citations
Ioan Raicu, Yong Zhao, Cătălin Dumitrescu et al. · 2007 · 306 citations