Mahout: Low-overhead datacenter traffic management using end-host-based elephant detection

TLDR

Datacenters rely on high‑bandwidth interconnects and redundant topologies, yet effective traffic management is required to exploit bisection bandwidth, and existing approaches suffer from high monitoring overhead, switch resource consumption, and long detection times. The authors aim to detect and manage large elephant flows at end hosts using a low‑overhead system called Mahout. Mahout monitors end‑host socket buffers to detect elephant flows and then signals a central controller via low‑overhead in‑band messaging, following an OpenFlow‑like architecture. Analytical evaluation and experiments demonstrate that Mahout outperforms previous solutions in detection speed and resource efficiency.

Abstract

Datacenters need high-bandwidth interconnection fabrics. Several researchers have proposed highly-redundant topologies with multiple paths between pairs of end hosts for datacenter networks. However, traffic management is necessary to effectively utilize the bisection bandwidth provided by these topologies. This requires timely detection of elephant flows-flows that carry large amount of data-and managing those flows. Previously proposed approaches incur high monitoring overheads, consume significant switch resources, and/or have long detection times. We propose, instead, to detect elephant flows at the end hosts. We do this by observing the end hosts's socket buffers, which provide better, more efficient visibility of flow behavior. We present Mahout, a low-overhead yet effective traffic management system that follows OpenFlow-like central controller approach for network management but augments the design with our novel end host mechanism. Once an elephant flow is detected, an end host signals the network controller using in-band signaling with low overheads. Through analytical evaluation and experiments, we demonstrate the benefits of Mahout over previous solutions.