Towards Automatic Concept-based Explanations

TLDR

Interpretability has become a critical research area as machine learning models increasingly influence important decisions, yet existing methods mainly provide per‑sample feature importance scores, and summarizing these scores into coherent, dataset‑wide insights remains challenging. This study proposes principles and desiderata for concept‑based explanations that move beyond individual feature importance to identify higher‑level, human‑understandable concepts applicable across the entire dataset. To realize this, the authors develop ACE, an algorithm that automatically extracts visual concepts from data. Experiments show that ACE discovers concepts that are human‑meaningful, coherent, and crucial for the neural network’s predictions.

Abstract

Interpretability has become an important topic of research as more machine learning (ML) models are deployed and widely used to make important decisions. Most of the current explanation methods provide explanations through feature importance scores, which identify features that are important for each individual input. However, how to systematically summarize and interpret such per sample feature importance scores itself is challenging. In this work, we propose principles and desiderata for \emph{concept} based explanation, which goes beyond per-sample features to identify higher-level human-understandable concepts that apply across the entire dataset. We develop a new algorithm, ACE, to automatically extract visual concepts. Our systematic experiments demonstrate that \alg discovers concepts that are human-meaningful, coherent and important for the neural network's predictions.