2011 · 61 citations · 15 references
ComputingComputer ScienceVisualizationMachine LearningStatistical LearningVisual AnalyticsClassificationLoss FunctionPattern RecognitionProbabilistic Machine LearningProbabilistic VisualizationStatistical VisualizationBrier ScoreClassification Accuracy MetricsClassification Error VisualizationClassification LossDiagnostic Performance MetricsOperating ConditionsProbabilistic Loss EstimationProbabilistic Loss Modeling
It is often necessary to evaluate classifier performance over a range of operating conditions, rather than as a point estimate. This is typically assessed through the construction of 'curves' over a 'space', visualising how one or two performance metrics vary with the operating condition. For binary classifiers in particular, cost space is a natural way of showing this range of performance, visualising loss against operating condition. However, the curves which have been traditionally drawn in cost space, known as cost curves, show the optimal loss, and hence assume knowledge of the optimal decision threshold for a given operating condition. Clearly, this leads to an optimistic assessment of classifier performance. In this paper we propose a more natural way of visualising classifier performance in cost space, which is to plot probabilistic loss on the y-axis, i.e., the loss arising from the probability estimates. This new curve provides new ways of understanding classifier performance and new tools to compare classifiers. In addition, we show that the area under this curve is exactly the Brier score, one of the most popular performance metrics for probabilistic classifiers.
15
UCI Machine Learning Repository
Arthur Asuncion · Medical Entomology and Zoology · 2007 · 24.4K citations
An introduction to ROC analysis
Tom Fawcett · Pattern Recognition Letters · 2005 · 21.3K citations
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
Glenn W. Brier · Monthly Weather Review · 1950 · 5.3K citations
Business, Mathematics, Meteorology +10
The foundations of cost-sensitive learning
Charles Elkan · 2001 · 1.8K citations
The DET curve in assessment of detection task performance
Alvin F. Martín, George R. Doddington, Terri Kamm et al. · 1997 · 1.4K citations