Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff

TLDR

AI systems are increasingly deployed to support human decision making in high‑stakes domains such as healthcare and criminal justice, where teams rely on the AI’s inferences; successful partnership requires humans to understand AI performance and failures, yet updates that improve predictive accuracy can alter behavior and erode prior user confidence. The study investigates how updates to an AI system affect human‑AI team performance and introduces the concept of update compatibility with prior user experience. The authors propose a re‑training objective that penalizes new errors to enhance the compatibility of AI updates with prior user experience. Empirical results on three high‑stakes classification tasks reveal that performance‑improving updates can hurt team performance, that current algorithms produce incompatible updates, and that the proposed objective can balance the performance/compatibility tradeoff to yield more compatible yet accurate updates.

Abstract

AI systems are being deployed to support human decision making in high-stakes domains such as healthcare and criminal justice. In many cases, the human and AI form a team, in which the human makes decisions after reviewing the AI’s inferences. A successful partnership requires that the human develops insights into the performance of the AI system, including its failures. We study the influence of updates to an AI system in this setting. While updates can increase the AI’s predictive performance, they may also lead to behavioral changes that are at odds with the user’s prior experiences and confidence in the AI’s inferences. We show that updates that increase AI performance may actually hurt team performance. We introduce the notion of the compatibility of an AI update with prior user experience and present methods for studying the role of compatibility in human-AI teams. Empirical results on three high-stakes classification tasks show that current machine learning algorithms do not produce compatible updates. We propose a re-training objective to improve the compatibility of an update by penalizing new errors. The objective offers full leverage of the performance/compatibility tradeoff across different datasets, enabling more compatible yet accurate updates.