Symmetric Cross Entropy for Robust Learning With Noisy Labels

TLDR

Training deep neural networks with noisy labels is challenging, and existing methods leave unresolved issues, particularly in handling hard classes that require noise‑tolerant terms. The authors aim to demonstrate that cross‑entropy overfits easy classes and underlearns hard ones, and to propose Symmetric Cross Entropy (SL) that simultaneously mitigates both problems. SL augments standard cross‑entropy with a reverse cross‑entropy term derived from symmetric KL‑divergence to provide noise‑robust learning. Theoretical analysis and experiments on benchmark and real‑world datasets show SL outperforms state‑of‑the‑art methods.

Abstract

Training accurate deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Though a number of approaches have been proposed for learning with noisy labels, many open issues remain. In this paper, we show that DNN learning with Cross Entropy (CE) exhibits overfitting to noisy labels on some classes ("easy" classes), but more surprisingly, it also suffers from significant under learning on some other classes ("hard" classes). Intuitively, CE requires an extra term to facilitate learning of hard classes, and more importantly, this term should be noise tolerant, so as to avoid overfitting to noisy labels. Inspired by the symmetric KL-divergence, we propose the approach of Symmetric cross entropy Learning (SL), boosting CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). Our proposed SL approach simultaneously addresses both the under learning and overfitting problem of CE in the presence of noisy labels. We provide a theoretical analysis of SL and also empirically show, on a range of benchmark and real-world datasets, that SL outperforms state-of-the-art methods. We also show that SL can be easily incorporated into existing methods in order to further enhance their performance.