Known Unknowns: A Critical Determinant of Confidence and Calibration

Abstract

We propose that an important determinant of judged confidence is the evaluation of evidence that is unknown or missing, and overconfidence is often driven by the neglect of unknowns. We contrast this account with prior research suggesting that overconfidence is due to biased processing of known evidence in favor of a focal hypothesis. In Study 1, we asked participants to list their thoughts as they answered two-alternative forced-choice trivia questions and judged the probability that their answers were correct. Participants who thought more about unknowns were less overconfident. In Studies 2 and 3, we asked participants to list unknowns before assessing their confidence. “Considering the unknowns” reduced overconfidence substantially and was more effective than the classic “consider the alternative” debiasing technique. Moreover, considering the unknowns selectively reduced confidence in domains where participants were overconfident but did not affect confidence in domains where participants were well-calibrated or underconfident. Data, as supplemental material, are available at https://doi.org/10.1287/mnsc.2016.2580 . This paper was accepted by Yuval Rottenstreich, judgment and decision making.