Invariance Principle Meets Information Bottleneck for\n Out-of-Distribution Generalization

Abstract

The invariance principle from causality is at the heart of notable approaches\nsuch as invariant risk minimization (IRM) that seek to address\nout-of-distribution (OOD) generalization failures. Despite the promising\ntheory, invariance principle-based approaches fail in common classification\ntasks, where invariant (causal) features capture all the information about the\nlabel. Are these failures due to the methods failing to capture the invariance?\nOr is the invariance principle itself insufficient? To answer these questions,\nwe revisit the fundamental assumptions in linear regression tasks, where\ninvariance-based approaches were shown to provably generalize OOD. In contrast\nto the linear regression tasks, we show that for linear classification tasks we\nneed much stronger restrictions on the distribution shifts, or otherwise OOD\ngeneralization is impossible. Furthermore, even with appropriate restrictions\non distribution shifts in place, we show that the invariance principle alone is\ninsufficient. We prove that a form of the information bottleneck constraint\nalong with invariance helps address key failures when invariant features\ncapture all the information about the label and also retains the existing\nsuccess when they do not. We propose an approach that incorporates both of\nthese principles and demonstrate its effectiveness in several experiments.\n