Evaluating Mobile Apps with A/B and Quasi A/B Tests

TLDR

Mobile app usage has exploded, and while A/B testing is a standard evaluation framework widely used online, running such tests on mobile apps is challenging due to build, review, and release constraints, and many features cannot be A/B tested because of infrastructure and redesign issues. The paper aims to improve mobile app release evaluation by addressing infrastructure and user behavior differences and proposing quasi‑experimental techniques, illustrated with results from a recent LinkedIn launch. The authors employ randomized A/B tests for individual features and a quasi‑experimental framework to assess the overall app impact. They demonstrate the effectiveness of these quasi‑experimental techniques with data from a major LinkedIn app launch.

Abstract

We have seen an explosive growth of mobile usage, particularly on mobile apps. It is more important than ever to be able to properly evaluate mobile app release. A/B testing is a standard framework to evaluate new ideas. We have seen much of its applications in the online world across the industry [9,10,12]. Running A/B tests on mobile apps turns out to be quite different, and much of it is attributed to the fact that we cannot ship code easily to mobile apps other than going through a lengthy build, review and release process. Mobile infrastructure and user behavior differences also contribute to how A/B tests are conducted differently on mobile apps, which will be discussed in details in this paper. In addition to measuring features individually in the new app version through randomized A/B tests, we have a unique opportunity to evaluate the mobile app as a whole using the quasi-experimental framework [21]. Not all features can be A/B tested due to infrastructure changes and wholistic product redesign. We propose and establish quasi-experimental techniques for measuring impact from mobile app release, with results shared from a recent major app launch at LinkedIn.