Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules

Abstract

Strictly proper scoring rules continue to play an important role in probability assessment. Although many such rules have been developed, relatively little guidance exists as to which rule is the most appropriate. In this paper, we discuss two important properties of quadratic, spherical, and logarithmic scoring rules. From an ex post perspective, we compare their rank order properties and conclude that both quadratic and spherical scoring perform poorly in this regard, relative to logarithmic. Second, from an ex ante perspective, we demonstrate that in many situations, logarithmic scoring is the method least affected by a nonlinear utility function. These results suggest that logarithmic scoring is superior when rank order results are important and/or when the assessor has a nonlinear utility function. In addition to these results, and perhaps more important, we demonstrate that nonlinear utility induces relatively little deviation from the optimal assessment under an assumption of risk neutrality. These results provide both comfort and guidance to those who would like to use scoring rules as part of the assessment process.