Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings

TLDR

Software defect prediction seeks to enhance software quality by building classification models from code metrics, yet inconsistent findings across studies underscore the need for more reliable comparative research. To address these issues, the authors propose and apply a framework for large‑scale comparative experiments involving 22 classifiers on 10 public NASA datasets. The framework mitigates bias by examining three sources—limited datasets, inappropriate accuracy metrics, and insufficient statistical testing—and systematically compares classifier performance across the datasets. The results confirm that metric‑based classification is generally accurate, but reveal no significant performance differences among the top 17 classifiers, indicating that algorithm choice may be less critical than previously assumed.

Abstract

Software defect prediction strives to improve software quality and testing efficiency by constructing predictive classification models from code attributes to enable a timely identification of fault-prone modules. Several classification models have been evaluated for this task. However, due to inconsistent findings regarding the superiority of one classifier over another and the usefulness of metric-based classification in general, more research is needed to improve convergence across studies and further advance confidence in experimental results. We consider three potential sources for bias: comparing classifiers over one or a small number of proprietary data sets, relying on accuracy indicators that are conceptually inappropriate for software defect prediction and cross-study comparisons, and, finally, limited use of statistical testing procedures to secure empirical findings. To remedy these problems, a framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over 10 public domain data sets from the NASA Metrics Data repository. Overall, an appealing degree of predictive accuracy is observed, which supports the view that metric-based classification is useful. However, our results indicate that the importance of the particular classification algorithm may be less than previously assumed since no significant performance differences could be detected among the top 17 classifiers.