Long-Tailed Classification by Keeping the Good and Removing the Bad\n Momentum Causal Effect

Abstract

As the class size grows, maintaining a balanced dataset across many classes\nis challenging because the data are long-tailed in nature; it is even\nimpossible when the sample-of-interest co-exists with each other in one\ncollectable unit, e.g., multiple visual instances in one image. Therefore,\nlong-tailed classification is the key to deep learning at scale. However,\nexisting methods are mainly based on re-weighting/re-sampling heuristics that\nlack a fundamental theory. In this paper, we establish a causal inference\nframework, which not only unravels the whys of previous methods, but also\nderives a new principled solution. Specifically, our theory shows that the SGD\nmomentum is essentially a confounder in long-tailed classification. On one\nhand, it has a harmful causal effect that misleads the tail prediction biased\ntowards the head. On the other hand, its induced mediation also benefits the\nrepresentation learning and head prediction. Our framework elegantly\ndisentangles the paradoxical effects of the momentum, by pursuing the direct\ncausal effect caused by an input sample. In particular, we use causal\nintervention in training, and counterfactual reasoning in inference, to remove\nthe "bad" while keep the "good". We achieve new state-of-the-arts on three\nlong-tailed visual recognition benchmarks: Long-tailed CIFAR-10/-100,\nImageNet-LT for image classification and LVIS for instance segmentation.\n