Deep Residual Flow for Out of Distribution Detection

TLDR

Neural networks rely on accurate out‑of‑distribution detection, and current methods typically model feature‑activation distributions using Gaussian models. The study proposes a new out‑of‑distribution detection method that improves state‑of‑the‑art performance by employing an expressive density model based on normalizing flows. The method introduces a residual flow architecture that learns the residual distribution from a base Gaussian and can be applied to any approximately Gaussian data, demonstrated on ResNet and DenseNet models across image datasets. The approach achieves a principled improvement over the state‑of‑the‑art, raising the true negative rate from 56.7 % to 77.5 % on a ResNet trained on CIFAR‑100 when detecting ImageNet out‑of‑distribution samples at 95 % TPR.

Abstract

The effective application of neural networks in the real-world relies on proficiently detecting out-of-distribution examples. Contemporary methods seek to model the distribution of feature activations in the training data for adequately distinguishing abnormalities, and the state-of-the-art method uses Gaussian distribution models. In this work, we present a novel approach that improves upon the state-of-the-art by leveraging an expressive density model based on normalizing flows. We introduce the residual flow, a novel flow architecture that learns the residual distribution from a base Gaussian distribution. Our model is general, and can be applied to any data that is approximately Gaussian. For out of distribution detection in image datasets, our approach provides a principled improvement over the state-of-the-art. Specifically, we demonstrate the effectiveness of our method in ResNet and DenseNet architectures trained on various image datasets. For example, on a ResNet trained on CIFAR-100 and evaluated on detection of out-of-distribution samples from the ImageNet dataset, holding the true positive rate (TPR) at 95%, we improve the true negative rate (TNR) from 56.7% (current state of-the-art) to 77.5% (ours).