EngineeringComputer EngineeringComputer SystemsComputer NetworksEmbedded SystemsInterconnection NetworksDirect Memory AccessInterconnect ArchitectureMemory Hierarchy DesignMemory Subsystem DesignNetwork ArchitectureRemote Memory AccessAddressingData Center Network DesignDirect Memory Access ArchitectureEmpirical EvaluationHardware ResourcesRemote Direct Memory AccessZero-copy CommunicationZero-copy Memory Access
Disaggregation of hardware resources that are traditionally embedded within individual servers into separate resource pools is an emerging architectural trend in hyperscale data center design, as exemplified by Facebook's disaggregated rack architecture. This paper presents the design, implementation and evaluation of a PCIe-based rack area network system called Marlin, which is designed to support the communications needs of disaggregated racks. By virtue of being based on PCIe, Marlin presents a memory-based addressing model for both I/O device sharing among multiple hosts and inter-host communications, and as a result offers hardware- based remote direct memory access (HRDMA) as a first-class communications primitive between servers within a rack. Marlin supports socket-based communications for legacy network applications and cross-machine zero memory copying for applications designed specifically to take full advantage of Marlin. Empirical measurements on a fully operational Marlin prototype based on 4-lane Gen3 PCIe technology show that the one-way kernel-to-kernel latency is 8.5μsec and the end-to- end sustainable TCP throughput is 19.6 Gbps.
20
Nick McKeown, Tom Anderson, Hari Balakrishnan et al. · ACM SIGCOMM Computer Communication Review · 2008 · 8.4K citations
Thorsten von Eicken, Anindya Basu, V. Buch et al. · 1995 · 719 citations
Disaggregated memory for expansion and sharing in blade servers
Kevin Lim, Jichuan Chang, Trevor Mudge et al. · 2009 · 365 citations
IntelŴVirtualization Technology for Directed I/O
Darren Abramson · Intel Technology Journal · 2006 · 299 citations
The Virtual Interface Architecture
Dave Dunning, G. Regnier, G. McAlpine et al. · IEEE Micro · 1998 · 291 citations