The information bottleneck method

TLDR

Relevant information in a signal \(x\) is defined as the information it provides about another signal \(y\), such as face images about names or speech sounds about words, and understanding \(x\) requires identifying which features of \(\mathcal{X}\) contribute to predicting \(y\). The study aims to find a concise representation of \(\mathcal{X}\) that maximally preserves information about \(\mathcal{Y}\). The authors formulate a bottleneck optimization that generalizes rate‑distortion theory, deriving self‑consistent coding equations solved by a convergent re‑estimation method extending the Blahut–Arimoto algorithm. The method produces exact self‑consistent equations for the coding rules and offers a versatile variational framework applicable to diverse signal‑processing and learning problems.

Abstract

We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\X$ that preserves the maximum information about $\Y$. That is, we squeeze the information that $\X$ provides about $\Y$ through a `bottleneck' formed by a limited set of codewords $\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.