Quality and productivity outcomes relating to continuous integration in GitHub

TLDR

Software development involves multiple stages—coding, building, testing, deployment, and operations—and the integration and automation of these steps have long been a key concern, intensified by market demands for agility and decentralized collaboration. The study aims to determine whether continuous integration actually improves project performance by analyzing extensive historical data from GitHub projects. The authors analyze large-scale historical data from GitHub projects, examining process metrics and outcomes to assess the impact of continuous integration. Continuous integration increases team productivity by enabling more frequent integration of external contributions, without compromising code quality.

Abstract

Software processes comprise many steps; coding is followed by building, integration testing, system testing, deployment, operations, among others. Software process integration and automation have been areas of key concern in software engineering, ever since the pioneering work of Osterweil; market pressures for Agility, and open, decentralized, software development have provided additional pressures for progress in this area. But do these innovations actually help projects? Given the numerous confounding factors that can influence project performance, it can be a challenge to discern the effects of process integration and automation. Software project ecosystems such as GitHub provide a new opportunity in this regard: one can readily find large numbers of projects in various stages of process integration and automation, and gather data on various influencing factors as well as productivity and quality outcomes. In this paper we use large, historical data on process metrics and outcomes in GitHub projects to discern the effects of one specific innovation in process automation: continuous integration. Our main finding is that continuous integration improves the productivity of project teams, who can integrate more outside contributions, without an observable diminishment in code quality.