Limits to low-latency communication on high-speed networks

TLDR

Local area network throughput is rapidly increasing, with ATM and FDDI offering an order of magnitude higher bandwidth than Ethernet, and future technologies promising further gains, yet in distributed systems low latency remains the primary concern. The study investigates how newer high‑speed network technologies affect low‑latency, cross‑machine communication by comparing different hardware, network, and software designs. A new low‑latency remote procedure call system was designed and implemented, then ported to DECstation and SPARCstation platforms running ATM, FDDI, and Ethernet networks. The RPC system achieved substantially reduced call times—170 µs on an ATM network with DECstation 5000/200 hosts—demonstrating that processor advances and software design can bring small‑packet RPCs close to network limits, thereby highlighting the critical role of network and controller design in achieving true low latency.

Abstract

The throughput of local area networks is rapidly increasing. For example, the bandwidth of new ATM networks and FDDI token rings is an order of magnitude greater than that of Ethernets. Other network technologies promise a bandwidth increase of yet another order of magnitude in several years. However, in distributed systems, lowered latency rather than increased throughput is often of primary concern. This paper examines the system-level effects of newer high-speed network technologies on low-latency, cross-machine communications. To evaluate a number of influences, both hardware and software, we designed and implemented a new remote procedure call system targeted at providing low latency. We then ported this system to several hardware platforms (DECstation and SPARCstation) with several different networks and controllers (ATM, FDDI, and Ethernet). Comparing these systems allows us to explore the performance impact of alternative designs in the communication system with respect to achieving low latency, e.g., the network, the network controller, the hose architecture and cache system, and the kernel and user-level runtime software. Our RPC system, which achieves substantially reduced call times (170 μseconds on an ATM network using DECstation 5000/200 hosts), allows us to isolate those components of next-generation networks and controllers that still stand in the way of low-latency communication. We demonstrate that new-generation processor technology and software design can reduce small-packet RPC times to near network-imposed limits, making network and controller design more crucial than ever to achieving truly low-latency communication.