Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks

TLDR

This paper investigates how people decide to rely on AI decision aids when objective performance data is scarce. The authors conduct three randomized experiments to examine the heuristics people use to adjust reliance on machine learning models under limited performance feedback. They find that high confidence agreement between humans and the model increases reliance when no performance data is available, but this effect changes once aggregate model performance is disclosed, and that the influence of high confidence agreement is moderated by human confidence in disagreement cases, highlighting risks and informing design to promote appropriate reliance.

Abstract

This paper addresses an under-explored problem of AI-assisted decision-making: when objective performance information of the machine learning model underlying a decision aid is absent or scarce, how do people decide their reliance on the model? Through three randomized experiments, we explore the heuristics people may use to adjust their reliance on machine learning models when performance feedback is limited. We find that the level of agreement between people and a model on decision-making tasks that people have high confidence in significantly affects reliance on the model if people receive no information about the model's performance, but this impact will change after aggregate-level model performance information becomes available. Furthermore, the influence of high confidence human-model agreement on people's reliance on a model is moderated by people's confidence in cases where they disagree with the model. We discuss potential risks of these heuristics, and provide design implications on promoting appropriate reliance on AI.