From Infrastructure to Culture

TLDR

A/B testing is a widely adopted method for evaluating user engagement on online platforms, with LinkedIn now running over 400 concurrent experiments daily and prior work outlining its frameworks and challenges. This paper details LinkedIn’s experimentation platform and the specific challenges of conducting large‑scale A/B tests in a social‑network context. The platform is described from experiment design and deployment to analysis, including advanced scenarios such as offline tests and network‑effect mitigation, and outlines key features and processes for fostering a robust experimentation culture.

Abstract

A/B testing, also known as bucket testing, split testing, or controlled experiment, is a standard way to evaluate user engagement or satisfaction from a new service, feature, or product. It is widely used among online websites, including social network sites such as Facebook, LinkedIn, and Twitter to make data-driven decisions. At LinkedIn, we have seen tremendous growth of controlled experiments over time, with now over 400 concurrent experiments running per day. General A/B testing frameworks and methodologies, including challenges and pitfalls, have been discussed extensively in several previous KDD work [7, 8, 9, 10]. In this paper, we describe in depth the experimentation platform we have built at LinkedIn and the challenges that arise particularly when running A/B tests at large scale in a social network setting. We start with an introduction of the experimentation platform and how it is built to handle each step of the A/B testing process at LinkedIn, from designing and deploying experiments to analyzing them. It is then followed by discussions on several more sophisticated A/B testing scenarios, such as running offline experiments and addressing the network effect, where one user's action can influence that of another. Lastly, we talk about features and processes that are crucial for building a strong experimentation culture.