Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models

TLDR

Defect prediction models enable software quality teams to allocate resources to the most defect‑prone modules, yet prior NASA dataset studies found classification technique had little effect, a conclusion that may be due to the dataset’s noise and bias. The study aims to replicate that analysis in two experimental settings. The authors replicated the prior procedure on the noisy NASA dataset, confirming minimal technique impact, and then applied it to a cleaned NASA dataset and the PROMISE dataset of diverse open‑source projects. In the cleaned NASA and PROMISE datasets, classification technique choice significantly affects defect prediction performance, with some techniques outperforming others, contrary to earlier findings.

Abstract

Defect prediction models help software quality assurance teams to effectively allocate their limited resources to the most defect-prone software modules. A variety of classification techniques have been used to build defect prediction models ranging from simple (e.g., Logistic regression) to advanced techniques (e.g., Multivariate Adaptive Regression Splines (MARS)). Surprisingly, recent research on the NASA dataset suggests that the performance of a defect prediction model is not significantly impacted by the classification technique that is used to train it. However, the dataset that is used in the prior study is both: (a) noisy, i.e., Contains erroneous entries and (b) biased, i.e., Only contains software developed in one setting. Hence, we set out to replicate this prior study in two experimental settings. First, we apply the replicated procedure to the same (known-to-be noisy) NASA dataset, where we derive similar results to the prior study, i.e., The impact that classification techniques have appear to be minimal. Next, we apply the replicated procedure to two new datasets: (a) the cleaned version of the NASA dataset and (b) the PROMISE dataset, which contains open source software developed in a variety of settings (e.g., Apache, GNU). The results in these new datasets show a clear, statistically distinct separation of groups of techniques, i.e., The choice of classification technique has an impact on the performance of defect prediction models. Indeed, contrary to earlier research, our results suggest that some classification techniques tend to produce defect prediction models that outperform others.