How do people react to AI failure? Automation bias, algorithmic aversion, and perceived controllability

Abstract

Abstract AI can make mistakes and cause unfavorable consequences. It is important to know how people react to such AI-driven negative consequences and subsequently evaluate the fairness of AI’s decisions. This study theorizes and empirically tests two psychological mechanisms that explain the process: (a) heuristic expectations of AI’s consistent performance (automation bias) and subsequent frustration of unfulfilled expectations (algorithmic aversion) and (b) heuristic perceptions of AI’s controllability over negative results. Our findings from two experimental studies reveal that these two mechanisms work in an opposite direction. First, participants tend to display more sensitive responses to AI’s inconsistent performance and thus make more punitive assessments of AI’s decision fairness, when compared to responses to human experts. Second, as participants perceive AI has less control over unfavorable outcomes than human experts, they are more tolerant in their assessments of AI.