A Framework for Automating the Measurement of DevOps Research and Assessment (DORA) Metrics

Abstract

The DevOps Research and Assessment (DORA) metrics have been widely accepted by the software industry as a powerful method to quantify DevOps performance, leading to significant interest in their measurement. Existing proprietary solutions are highly customised, and require specific types of cloud infrastructure, limiting their suitability for projects such as libraries, frameworks, and open source projects. To address this gap, we present a framework which operationalizes the DORA metrics independently of a project’s software development life-cycle or type of deployment. We demonstrate the general applicability of this framework by using it to calculate the throughput and stability of 304 popular open source repositories. We find that the time-series data it produces provides meaningful insights into the trending direction of a project’s recent and retrospective throughput and stability performance, especially when significant changes in metrics are correlated with major events in the project’s history. We conclude with recommendations for augmenting our approach with additional information such as bug criticality when such information is available.