On the ROC Score of Probability Forecasts

Abstract

The relative operating characteristic (ROC) is a measure of the quality of probability forecasts that relates the hit rate to the corresponding false-alarm rate. This paper examines some aspects of the ROC curve for probability forecasts of three equiprobable categories (below normal, near normal, and above normal) in the framework of a simple analog of a climate forecasting system. The insensitivity of the ROC score to some types of forecast biases is discussed and the link to deterministic potential predictability is established in the context of the simple forecasting system. The findings are illustrated with a collection of 24-member ensemble hindcasts of seasonal mean 700-hPa temperature produced with the second-generation general circulation model of the Canadian Centre for Climate Modelling and Analysis.