Mesos: a platform for fine-grained resource sharing in the data center

TLDR

Mesos is a platform that enables sharing commodity clusters among diverse cluster computing frameworks like Hadoop and MPI. Mesos implements fine‑grained resource sharing via a distributed two‑level scheduling mechanism that offers resources to frameworks, which then choose resources and tasks to maintain data locality. Mesos improves cluster utilization, avoids per‑framework data replication, achieves near‑optimal data locality, scales to 50,000 nodes, and remains resilient to failures.

Abstract

We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today's frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our results show that Mesos can achieve near-optimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) nodes, and is resilient to failures.