Evaluating Performance of Software Defect Prediction Models Using Area Under Precision-Recall Curve (AUC-PR)

Abstract

Software defect prediction (SDP) models are used to improve effort and testing estimate of software by identifying defective modules beforehand. Precision, recall/true positive rate and false positive rate have been used to evaluate the performance of models. In literature, area under receiver operating characteristic curve (AUC-ROC) has been used to evaluate the model performance. The standard learning goal of the defect model is to optimize the (AUC-ROC). Use of this measure has also been advocated in numerous benchmarking studies. The literature has discussed the performance bar (or so-called ceiling effect) of AUC-ROC targeted models. The literature has also indicated the use of area under precision recall curve (AUC-PR) as an evaluation parameter for the models. This study investigates if AUC-PR curve gives different information regarding model performance. To this end this study ranks the existing models based on AUC-ROC and AUC-PR and report the change in ranking of these models. The change in ranking gives an opportunity to study if the ceiling effect can be managed and AUC-PR (instead of AUC-ROC) can be considered as a goal for the prediction models. AUC-PR based evaluation of the models can help avoid the extra cost, time, and effort employed to test non-defective modules.