Trustworthy online controlled experiments

TLDR

Online controlled experiments are widely used by major tech companies, and large‑scale deployment has revealed practical challenges that differ from theoretical expectations. The paper shares real‑world lessons from puzzling outcomes of online controlled experiments, aiming to explain their causes. The authors analyze each anomaly over weeks to months, covering topics such as overall evaluation criteria, click tracking, effect trends, experiment length, power, and carryover effects. The study finds that root causes of anomalous results are widespread, and that increased awareness and analysis of anomalies can significantly improve the trustworthiness and financial impact of online experiments.

Abstract

Online controlled experiments are often utilized to make data-driven decisions at Amazon, Microsoft, eBay, Facebook, Google, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher's experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale--thousands of experiments now--has taught us many lessons. These exemplify the proverb that the difference between theory and practice is greater in practice than in theory. We present our learnings as they happened: puzzling outcomes of controlled experiments that we analyzed deeply to understand and explain. Each of these took multiple-person weeks to months to properly analyze and get to the often surprising root cause. The root causes behind these puzzling results are not isolated incidents; these issues generalized to multiple experiments. The heightened awareness should help readers increase the trustworthiness of the results coming out of controlled experiments. At Microsoft's Bing, it is not uncommon to see experiments that impact annual revenue by millions of dollars, thus getting trustworthy results is critical and investing in understanding anomalies has tremendous payoff: reversing a single incorrect decision based on the results of an experiment can fund a whole team of analysts. The topics we cover include: the OEC (Overall Evaluation Criterion), click tracking, effect trends, experiment length and power, and carryover effects.