Cost-Sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs?

Gary M. Weiss, Kate McCarthy, Bibi Zabar

DMIN · 2007 · 229 citations · 15 references

Concepts

TL;DR

Classifiers trained on highly skewed data tend to favor majority classes, which is problematic in contexts such as medical diagnosis where minority class misclassification incurs a higher cost. The study compares three approaches—cost‑sensitive learning, oversampling, and undersampling—to determine which yields the best classifier under skewed distributions and unequal misclassification costs. The authors evaluate these methods by training classifiers on imbalanced data and measuring performance across different imbalance levels and cost ratios. The empirical comparison shows that the optimal method depends on the degree of imbalance and the relative misclassification costs, with cost‑sensitive learning often outperforming sampling techniques in high‑cost minority scenarios.

Abstract

The classifier built from a data set with a highly skewed class distribution generally predicts the more frequently occurring classes much more often than the infrequently occurring classes. This is largely due to the fact that most classifiers are designed to maximize accuracy. In many instances, such as for medical diagnosis, this classification behavior is unacceptable because the minority class is the class of primary interest (i.e., it has a much higher misclassification cost than the majority class). In this paper we compare three methods for dealing with data that has a skewed class distribution and nonuniform misclassification costs. The first method incorporates the misclassification costs into the learning algorithm while the other two methods employ oversampling or undersampling to make the training data more balanced. In this paper we empirically compare the effectiveness of these methods in order to determine which produces the best overall classifier—and under what circumstances.

References

15