A Theoretical Explanation for Perplexing Behaviors of\n Backpropagation-based Visualizations

Abstract

Backpropagation-based visualizations have been proposed to interpret\nconvolutional neural networks (CNNs), however a theory is missing to justify\ntheir behaviors: Guided backpropagation (GBP) and deconvolutional network\n(DeconvNet) generate more human-interpretable but less class-sensitive\nvisualizations than saliency map. Motivated by this, we develop a theoretical\nexplanation revealing that GBP and DeconvNet are essentially doing (partial)\nimage recovery which is unrelated to the network decisions. Specifically, our\nanalysis shows that the backward ReLU introduced by GBP and DeconvNet, and the\nlocal connections in CNNs are the two main causes of compelling visualizations.\nExtensive experiments are provided that support the theoretical analysis.\n