Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations

TLDR

Expressive classifiers such as neural networks are among the most accurate supervised learning methods in use today, but their opaque decision boundaries make them difficult to trust in critical applications. We propose a method to explain the predictions of any differentiable model via the gradient of the class label with respect to the input, which provides a normal to the decision boundary. The method uses this gradient to identify input dimensions of high sensitivity, is orders of magnitude faster than sample‑based perturbation methods such as LIME, and enables efficient discovery of multiple qualitatively different and expert‑consistent decision boundaries. Across multiple datasets, the approach generalizes much better when test conditions differ from those in training.

Abstract

Expressive classifiers such as neural networks are among the most accurate supervised learning methods in use today, but their opaque decision boundaries make them difficult to trust in critical applications. We propose a method to explain the predictions of any differentiable model via the gradient of the class label with respect to the input (which provides a normal to the decision boundary). Not only is this approach orders of magnitude faster at identifying input dimensions of high sensitivity than sample-based perturbation methods (e.g. LIME), but it also lends itself to efficiently discovering multiple qualitatively different decision boundaries as well as decision boundaries that are consistent with expert annotation. On multiple datasets, we show our approach generalizes much better when test conditions differ from those in training.