Manipulating and Measuring Model Interpretability

TLDR

Machine learning models are increasingly used in high‑stakes decision making, yet experimental evidence on whether interpretable models actually improve user behavior or error detection is scarce. We conducted three pre‑registered experiments with 800 participants, comparing models that differed only in feature count and transparency (clear vs black‑box). Clear, low‑feature models improved participants’ ability to simulate predictions but did not increase adherence, and actually impaired error detection, underscoring that intuition about interpretability can be misleading.

Abstract

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model's predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N = 3, 800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model's predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model's sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.