Training Normalizing Flows with the Information Bottleneck for\n Competitive Generative Classification

Abstract

The Information Bottleneck (IB) objective uses information theory to\nformulate a task-performance versus robustness trade-off. It has been\nsuccessfully applied in the standard discriminative classification setting. We\npose the question whether the IB can also be used to train generative\nlikelihood models such as normalizing flows. Since normalizing flows use\ninvertible network architectures (INNs), they are information-preserving by\nconstruction. This seems contradictory to the idea of a bottleneck. In this\nwork, firstly, we develop the theory and methodology of IB-INNs, a class of\nconditional normalizing flows where INNs are trained using the IB objective:\nIntroducing a small amount of {\\em controlled} information loss allows for an\nasymptotically exact formulation of the IB, while keeping the INN's generative\ncapabilities intact. Secondly, we investigate the properties of these models\nexperimentally, specifically used as generative classifiers. This model class\noffers advantages such as improved uncertainty quantification and\nout-of-distribution detection, but traditional generative classifier solutions\nsuffer considerably in classification accuracy. We find the trade-off parameter\nin the IB controls a mix of generative capabilities and accuracy close to\nstandard classifiers. Empirically, our uncertainty estimates in this mixed\nregime compare favourably to conventional generative and discriminative\nclassifiers.\n