Improving the sensitivity of online controlled experiments by utilizing pre-experiment data

TLDR

Online controlled experiments drive data‑driven decisions at major tech firms, yet small metric differences can have huge business impact, and at Bing experiments routinely affect millions of dollars, so improving sensitivity enables more precise assessment, smaller sample sizes, or shorter durations. The study proposes CUPED, an approach that leverages pre‑experiment data to reduce metric variability and improve experimental sensitivity. CUPED adjusts post‑experiment metrics using pre‑experiment data to reduce variance. On Bing, CUPED reduces variance by about 50%, enabling the same statistical power with half the users or half the duration, and is broadly applicable, practical, and easy to implement.

Abstract

Online controlled experiments are at the heart of making data-driven decisions at a diverse set of companies, including Amazon, eBay, Facebook, Google, Microsoft, Yahoo, and Zynga. Small differences in key metrics, on the order of fractions of a percent, may have very significant business implications. At Bing it is not uncommon to see experiments that impact annual revenue by millions of dollars, even tens of millions of dollars, either positively or negatively. With thousands of experiments being run annually, improving the sensitivity of experiments allows for more precise assessment of value, or equivalently running the experiments on smaller populations (supporting more experiments) or for shorter durations (improving the feedback cycle and agility). We propose an approach (CUPED) that utilizes data from the pre-experiment period to reduce metric variability and hence achieve better sensitivity. This technique is applicable to a wide variety of key business metrics, and it is practical and easy to implement. The results on Bing's experimentation system are very successful: we can reduce variance by about 50%, effectively achieving the same statistical power with only half of the users, or half the duration.