Behaviour of verification measures for deterministic binary forecasts with respect to random changes and thresholding

Abstract

There are many possible verification measures when deterministic binary forecasts are made. In deciding which to use, various properties of the measures are of interest. One type of property explores what happens when certain types of change are made to the forecasts. In this article, two types of change and their effects on various verification measures are examined. Underlying some desirable properties of measures, such as propriety, equitability and consistency, is the desire that it should be impossible to improve the value, or expected value, of a measure by changing the original forecast from the forecaster's ‘true belief’. This behaviour is often called ‘hedging’. Among the types of change that ought not to lead to improved forecasts are random changes of subsets of the true binary forecasts. The effect of such changes is discussed here. A second type of change examined here arises when a continuous forecast is thresholded to produce the binary forecast. When this type of ‘thresholding’ is practiced a posteriori, it can be seen as a fine‐tuning of the forecasting system, in order to improve its verification performance. In the thresholding framework, one can optimize some verification performance measures with respect to others. In particular, this article discusses the optimization of bias and of the Peirce skill score.