An extensive comparison of bug prediction approaches

TLDR

Software defect prediction is a key goal in software engineering, yet the lack of a standard benchmark hampers comparison of the many existing approaches. The study introduces a publicly available benchmark dataset and compares the explanatory and predictive performance of established and novel bug‑prediction models. The authors compiled a benchmark dataset from multiple software systems and evaluated both existing and newly developed bug‑prediction models against it. The comparison revealed differences in performance and stability among the models, yielding several insights into bug‑prediction effectiveness.

Abstract

Reliably predicting software defects is one of software engineering's holy grails. Researchers have devised and implemented a plethora of bug prediction approaches varying in terms of accuracy, complexity and the input data they require. However, the absence of an established benchmark makes it hard, if not impossible, to compare approaches. We present a benchmark for defect prediction, in the form of a publicly available data set consisting of several software systems, and provide an extensive comparison of the explanative and predictive power of well-known bug prediction approaches, together with novel approaches we devised. Based on the results, we discuss the performance and stability of the approaches with respect to our benchmark and deduce a number of insights on bug prediction models.