<i>Why and why not</i> explanations improve the intelligibility of context-aware intelligent systems

TLDR

Context‑aware intelligent systems use implicit inputs and complex models that are often opaque, undermining user trust and acceptance. The study investigates whether automatically generated explanations can improve users’ understanding and trust of such systems. A controlled experiment with over 200 participants presented system operations and varied explanation types, then assessed user comprehension. Explanations that describe why the system behaved a certain way improved understanding and trust, whereas explanations of why it did not behave a certain way lowered understanding but maintained performance.

Abstract

Context-aware intelligent systems employ implicit inputs, and make decisions based on complex rules and machine learning models that are rarely clear to users. Such lack of system intelligibility can lead to loss of user trust, satisfaction and acceptance of these systems. However, automatically providing explanations about a system's decision process can help mitigate this problem. In this paper we present results from a controlled study with over 200 participants in which the effectiveness of different types of explanations was examined. Participants were shown examples of a system's operation along with various automatically generated explanations, and then tested on their understanding of the system. We show, for example, that explanations describing why the system behaved a certain way resulted in better understanding and stronger feelings of trust. Explanations describing why the system did not behave a certain way, resulted in lower understanding yet adequate performance. We discuss implications for the use of our findings in real-world context-aware applications.