Marlin

Abstract

Disaggregation of hardware resources that are traditionally embedded within individual servers into separate resource pools is an emerging architectural trend in hyperscale data center design, as exemplified by Facebook's disaggregated rack architecture. This paper presents the design, implementation and evaluation of a PCIe-based rack area network system called Marlin, which is designed to support the communications needs of disaggregated racks. By virtue of being based on PCIe, Marlin presents a memory-based addressing model for both I/O device sharing among multiple hosts and inter-host communications, and as a result offers hardware- based remote direct memory access (HRDMA) as a first-class communications primitive between servers within a rack. Marlin supports socket-based communications for legacy network applications and cross-machine zero memory copying for applications designed specifically to take full advantage of Marlin. Empirical measurements on a fully operational Marlin prototype based on 4-lane Gen3 PCIe technology show that the one-way kernel-to-kernel latency is 8.5μsec and the end-to- end sustainable TCP throughput is 19.6 Gbps.