On the impact of packet spraying in data center networks

TLDR

Data center networks use multi‑rooted tree topologies and equal‑cost multipath routing, which can cause load imbalance; splitting flows improves balance but is avoided due to concerns that packet reordering harms TCP. The study reexamines the belief that packet reordering harms TCP in data center networks with regular topologies. The authors evaluate random packet spraying on a real‑hardware data center testbed. Because equal‑cost paths are symmetric, TCP tolerates packet reordering from RPS and keeps a single RTT estimate, but performance degrades when symmetry is broken, prompting mitigation strategies.

Abstract

Modern data center networks are commonly organized in multi-rooted tree topologies. They typically rely on equal-cost multipath to split flows across multiple paths, which can lead to significant load imbalance. Splitting individual flows can provide better load balance, but is not preferred because of potential packet reordering that conventional wisdom suggests may negatively interact with TCP congestion control. In this paper, we revisit this "myth" in the context of data center networks which have regular topologies such as multi-rooted trees. We argue that due to symmetry, the multiple equal-cost paths between two hosts are composed of links that exhibit similar queuing properties. As a result, TCP is able to tolerate the induced packet reordering and maintain a single estimate of RTT. We validate the efficacy of random packet spraying (RPS) using a data center testbed comprising real hardware switches. We also reveal the adverse impact on the performance of RPS when the symmetry is disturbed (e.g., during link failures) and suggest solutions to mitigate this effect.