AUC Optimization vs. Error Rate Minimization

TLDR

The area under the ROC curve (AUC) is a common performance metric for classifiers, yet most algorithms optimize error rate rather than AUC. The study investigates the statistical relationship between AUC and error rate and evaluates RankBoost’s ability to globally optimize AUC. The authors derive exact expressions for the expected value and variance of AUC at a fixed error rate and analyze how these depend on classification accuracy. The results show that AUC increases monotonically with accuracy but its variability grows for imbalanced data and high error rates, and that RankBoost’s objective equals AUC under certain conditions, leading to better performance than algorithms that only approximate or locally optimize AUC.

Abstract

The area under an ROC curve (AUC) is a criterion used in many applications to measure the quality of a classification algorithm. However, the objective function optimized in most of these algorithms is the error rate and not the AUC value. We give a detailed statistical analysis of the relationship between the AUC and the error rate, including the first exact expression of the expected value and the variance of the AUC for a fixed error rate. Our results show that the average AUC is monotonically increasing as a function of the classification accuracy, but that the standard deviation for uneven distributions and higher error rates is noticeable. Thus, algorithms designed to minimize the error rate may not lead to the best possible AUC values. We show that, under certain conditions, the global function optimized by the RankBoost algorithm is exactly the AUC. We report the results of our experiments with RankBoost in several datasets demonstrating the benefits of an algorithm specifically designed to globally optimize the AUC over other existing algorithms optimizing an approximation of the AUC or only locally optimizing the AUC.