"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction

TLDR

Despite the proliferation of explainable AI methods, little is known about end‑users’ explainability needs and behaviors. The study aims to understand how explainability can support human‑AI interaction. We conducted a mixed‑methods study with 20 users of the Merlin bird‑identification app, gathering their XAI needs, uses, and perceptions. Participants wanted practically useful information to improve collaboration, used explanations to calibrate trust, enhance task skills, provide better inputs, give feedback, and preferred part‑based explanations resembling human reasoning.

Abstract

Despite the proliferation of explainable AI (XAI) methods, little is understood about end-users’ explainability needs and behaviors around XAI explanations. To address this gap and contribute to understanding how explainability can support human-AI interaction, we conducted a mixed-methods study with 20 end-users of a real-world AI application, the Merlin bird identification app, and inquired about their XAI needs, uses, and perceptions. We found that participants desire practically useful information that can improve their collaboration with the AI, more so than technical system details. Relatedly, participants intended to use XAI explanations for various purposes beyond understanding the AI’s outputs: calibrating trust, improving their task skills, changing their behavior to supply better inputs to the AI, and giving constructive feedback to developers. Finally, among existing XAI approaches, participants preferred part-based explanations that resemble human reasoning and explanations. We discuss the implications of our findings and provide recommendations for future XAI design.