Reliability and validity in comparative studies of software prediction models

TLDR

Empirical studies on software prediction models fail to converge on which model is best, and the reasons for this lack of consensus are poorly understood, especially as most compare machine learning to regression models. The study aims to develop more reliable research procedures to enable confident conclusions in comparative software prediction model studies. The authors conducted a simulation study comparing a machine learning and a regression model, evaluating the commonly used procedure of a single data sample, an accuracy indicator, and cross‑validation. The study finds that the standard research procedure is unreliable, likely driving the lack of convergence and undermining conclusions of comparative software prediction model studies.

Abstract

Empirical studies on software prediction models do not converge with respect to the question "which prediction model is best?" The reason for this lack of convergence is poorly understood. In this simulation study, we have examined a frequently used research procedure comprising three main ingredients: a single data sample, an accuracy indicator, and cross validation. Typically, these empirical studies compare a machine learning model with a regression model. In our study, we use simulation and compare a machine learning and a regression model. The results suggest that it is the research procedure itself that is unreliable. This lack of reliability may strongly contribute to the lack of convergence. Our findings thus cast some doubt on the conclusions of any study of competing software prediction models that used this research procedure as a basis of model comparison. Thus, we need to develop more reliable research procedures before we can have confidence in the conclusions of comparative studies of software prediction models.