Market-based Proportional Resource Sharing for Clusters

Abstract

Enabling technologies in high speed communication and global process scheduling have pushed clusters of computers into the mainstream as general-purpose high-performance computing systems. More generality, however, implies more sharing and this raises new questions in the area of cluster resource management. In particular, in systems where the aggregate demand for computing resources can exceed the aggregate supply, how to allocate resources amongst competing applications is an important problem. Traditional solutions to this problem have focused mainly on global optimization with respect to system-centric performance metrics, metrics which ignore higher level user intent. In this paper, we propose an alternative market-based approach based on the notion of a computational economy which optimizes for user value. Starting with fundamental requirements, we describe an abstract architecture for market-based cluster resource management based on the idea of proportional resource sharing of basic computing resources. Using this architecture, we have implemented a 32-node (64 processors) prototype system that provides a market for time-shared CPU usage for sequential and parallel programs. To begin evaluating our ideas, we are currently in the process of studying how users respond to the system by collecting data on real day-to-day usage of the cluster