The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale

TLDR

Software development companies increasingly aim to become data‑driven by continuously experimenting with customer‑facing products, yet they rarely succeed in adopting the methodology despite its competitive advantage. The paper aims to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations to become data‑driven at scale. The authors conduct an exhaustive collaborative case study in a large software‑intensive company and introduce the Experimentation Evolution Model, detailing technical, organizational, and business phases of scaling continuous controlled experimentation.

Abstract

Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution process of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the "Experimentation Evolution Model" in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale.