Conditional Probabilities, Relative Operating Characteristics, and Relative Operating Levels

TLDR

The ROC curve flexibly represents forecast quality for dichotomous, categorical, continuous, and probabilistic forecasts, and its use highlights the value of knowing event probabilities conditional on warnings. The study proposes an alternative to the ROC curve that quantifies forecast quality by estimating the probability of an event occurring given a warning, addressing construction issues and enabling skill optimization for continuous variables. The authors use ratios of events and nonevents associated with warnings to estimate probabilities of forewarned events and incorrect warnings, and extend this approach to construct a relative operating levels curve that identifies the threshold at which forecast skill is optimized.

Abstract

The relative operating characteristic (ROC) curve is a highly flexible method for representing the quality of dichotomous, categorical, continuous, and probabilistic forecasts. The method is based on ratios that measure the proportions of events and nonevents for which warnings were provided. These ratios provide estimates of the probabilities that an event will be forewarned and that an incorrect warning will be provided for a nonevent. Some guidelines for interpreting the ROC curve are provided. While the ROC curve is of direct interest to the user, the warning is provided in advance of the outcome and so there is additional value in knowing the probability of an event occurring contingent upon a warning being provided or not provided. An alternative method to the ROC curve is proposed that represents forecast quality when expressed in terms of probabilities of events occurring contingent upon the warnings provided. The ratios used provide estimates of the probability of an event occurring given the forecast that is issued. Some problems in constructing the curve in a manner that is directly analogous to that for the ROC curve are highlighted, and so an alternative approach is proposed. In the context of probabilistic forecasts, the ROC curve provides a means of identifying the forecast probability at which forecast value is optimized. In the context of continuous variables, the proposed relative operating levels curve indicates the exceedence threshold for defining an event at which forecast skill is optimized, and can enable the forecast user to estimate the probabilities of events other than that defined by the forecaster.