Performance testing of energy models: are we using the right statistical metrics?

Abstract

Testing the predictive performance of energy models (EMs) is necessary to evaluate their accuracies. This paper investigates the adequacy of existing statistical metrics that are often used by professionals and researchers to test EMs. It discerns that coefficient of variance of root mean squared error (CVRMSE) and mean bias error (MBE), which are prescribed in ASHRAE guideline 14, are not suitable for system-level energy model testing. It points out the limitations of CVRMSE, MBE, and also root mean squared error (RMSE). The analysis shows that the normalizing term of statistical metrics influences its accuracy in determining the predictive performance of EMs. An alternative metric (range normalized root mean squared error, RN_RMSE) is proposed that normalizes the RMSE by the range of the data as a replacement for CVRMSE. It is shown that RN_RMSE when used in tandem with can provide more meaningful and accurate representation of the performance of system-level EMs. Abbreviations Ems: energy models; IDFs: input data files; R2: coefficient of determination; RMSE: root mean squared error; CVRMSE: coefficient of variance of root mean squared error; MBE: mean bias error; RN RMSE: range normalized root mean squared error