Online Controlled Experimentation at Scale: An Empirical Survey on the Current State of A/B Testing

TLDR

Online controlled experiments (OCEs) are a powerful method for measuring feature value, used by companies such as Microsoft, Amazon, and Booking.com, but the maturity of the broader online software industry remains largely unknown. This study aims to assess the current state of A/B testing maturity across the software industry using a previously developed maturity model. The assessment is based on responses from 44 participants in an online empirical survey. Results indicate that companies typically build in‑house experimentation platforms of varying maturity, and that defining key metrics—overall evaluation criteria—remains the main challenge for successful experimentation.

Abstract

Online Controlled Experiments (OCEs, aka A/B tests) are one of the most powerful methods for measuring how much value new features and changes deployed to software products bring to users. Companies like Microsoft, Amazon, and Booking.com report the ability to conduct thousands of OCEs every year. However, the competences of the remainder of the online software industry remain unknown. The main objective of this paper is to reveal the current state of A/B testing maturity in the software industry based on a maturity model from our previous research. We base our findings on 44 responses from an online empirical survey. Our main contribution of this paper is the current state of experimentation maturity as operationalized by the ExG model for a convenience sample of companies doing online controlled experiments. Our findings show that, among others, companies typically develop in-house experimentation platforms, that these platforms are of various levels of maturity, and that designing key metrics - Overall Evaluation Criteria - remains the key challenge for successful experimentation.