Pip: detecting the unexpected in distributed systems

TLDR

Bugs in distributed systems are hard to find because they often arise from discrepancies between actual behavior and programmers' assumptions. The paper introduces Pip, an infrastructure that compares actual and expected behavior to expose structural errors and performance problems in distributed systems. Pip lets programmers declare expectations about communication, timing, and resource use, instruments the system to log actual behavior, and provides visualization and query tools to compare expectations with reality, with instrumentation and expectations automatically generated for applications such as FAB, SplitStream, Bullet, and RanSub. Pip enables developers to quickly understand and debug systems, automatically generates most instrumentation, makes writing expectations easy, and successfully identifies unexpected behavior and isolates causes of performance and correctness issues in all tested applications.

Abstract

Bugs in distributed systems are often hard to find. Many bugs reflect discrepancies between a system's behavior and the programmer's assumptions about that behavior. We present Pip, an infrastructure for comparing actual behavior and expected behavior to expose structural errors and performance problems in distributed systems. Pip allows programmers to express, in a declarative language, expectations about the system's communications structure, timing, and resource consumption. Pip includes system instrumentation and annotation tools to log actual system behavior, and visualization and query tools for exploring expected and unexpected behavior. Pip allows a developer to quickly understand and debug both familiar and unfamiliar systems. We applied Pip to several applications, including FAB, SplitStream, Bullet, and RanSub. We generated most of the instrumentation for all four applications automatically. We found the needed expectations easy to write, starting in each case with automatically generated expectations. Pip found unexpected behavior in each application, and helped to isolate the causes of poor performance and incorrect behavior.