Overconfidence in Interval Estimates.

TLDR

The study aims to examine the psychological mechanisms that explain why judges’ interval estimates are overconfident. Judges were asked to provide numerical lower and upper bounds for factual questions, such as the year of the first hot‑air balloon flight. Judges’ intervals were markedly overconfident, with the true answer falling inside the reported bounds far less often than the stated confidence level, and this overconfidence varied by elicitation method, domain, and gender.

Abstract

Judges were asked to make numerical estimates (e.g., "In what year was the first flight of a hot air balloon?") Judges provided high and low estimates such that they were X% sure that the correct answer lay between them. They exhibited substantial overconfidence: The correct answer fell inside their intervals much less than X% of the time. This contrasts with choices between 2 possible answers to a question, which showed much less overconfidence. The authors show that overconfidence in interval estimates can result from variability in setting interval widths. However, the main cause is that subjective intervals are systematically too narrow given the accuracy of one's information-sometimes only 40% as large as necessary to be well calibrated. The degree of overconfidence varies greatly depending on how intervals are elicited. There are also substantial differences among domains and between male and female judges. The authors discuss the possible psychological mechanisms underlying this pattern of findings.