Separating the Brier Score into Calibration and Refinement Components: A Graphical Exposition

Abstract

Abstract Proper scoring rules of subjective probability assessments have been shown to be separable into distinct calibration and refinement components. This article presents a graphical description of this separation theorem as applied to the Brier score (quadratic loss) of assessed probabilities for a sequence of observable events. Configurations of achievable calibration, refinement, and Brier scores are exhibited in three-dimensional space and by projection into interpretable subspaces. Relationships of calibration and refinement to the usual sum-of-squares partition in analysis of variance are denoted. Controversy concerning the implications of long-run calibration for probability theory based solely on the principle of coherence is outlined.