Explainable AI lacks regulative reasons: why AI and human decision-making are not equally opaque

TLDR

Many AI systems used for decision‑making are opaque, and some scholars argue that human decision‑making is equally opaque, so simplified reason‑giving explanations should be adequate for both. This paper argues that the claim that human and algorithmic decision‑making are equally opaque ignores the fact that human decision‑making can be more transparent and trustworthy. Because people who explain their decisions by giving reasons are often prompted to self‑regulate and act in ways that confirm those reason reports. AI explanation systems lack this self‑regulative feature, and ignoring this difference can lead to underestimating human transparency and to the development of misleading explainable AI.

Abstract

Abstract Many artificial intelligence (AI) systems currently used for decision-making are opaque, i.e., the internal factors that determine their decisions are not fully known to people due to the systems’ computational complexity. In response to this problem, several researchers have argued that human decision-making is equally opaque and since simplifying, reason-giving explanations (rather than exhaustive causal accounts) of a decision are typically viewed as sufficient in the human case, the same should hold for algorithmic decision-making. Here, I contend that this argument overlooks that human decision-making is sometimes significantly more transparent and trustworthy than algorithmic decision-making. This is because when people explain their decisions by giving reasons for them, this frequently prompts those giving the reasons to govern or regulate themselves so as to think and act in ways that confirm their reason reports. AI explanation systems lack this self-regulative feature. Overlooking it when comparing algorithmic and human decision-making can result in underestimations of the transparency of human decision-making and in the development of explainable AI that may mislead people by activating generally warranted beliefs about the regulative dimension of reason-giving.