Understanding and dealing with operator mistakes in internet services

TLDR

Operator mistakes are a significant source of unavailability in modern Internet services. The paper characterizes operator mistakes through extensive experiments on a realistic three‑tier auction service and proposes validating operator actions before they affect the system. We conduct experiments on a realistic three‑tier auction service and build a validation environment that extends the online system, allowing components to be validated with real workloads before migration. Observed operator mistakes include software misconfiguration, fault misdiagnosis, and incorrect software restarts, and the prototype validation system detects 66 % of them.

Abstract

Operator mistakes are a significant source of unavailability in modern Internet services. In this paper, we first characterize these mistakes by performing an extensive set of experiments using human operators and a realistic three-tier auction service. The mistakes we observed range from software misconfiguration, to fault misdiagnosis, to incorrect software restarts. We next propose to validate operator actions before they are made visible to the rest of the system. We demonstrate how to accomplish this task via the creation of a validation environment that is an extension of the online system, where components can be validated using real workloads before they are migrated into the running service. We show that our prototype validation system can detect 66% of the operator mistakes that we have observed.