Understanding the Replication Crisis as a Base Rate Fallacy

TLDR

The replication crisis in social psychology and clinical medicine stems from many well‑confirmed results being overturned, largely due to questionable research practices, publication bias, poor incentives, and fraud. The article argues that the high failure rate of replications can be explained by high‑quality science and proposes lessons for future research. The authors explain that a high proportion of false hypotheses leads to many seemingly supported results because well‑conducted experiments with a 5 % type‑I error rate will frequently produce false positives. The article concludes that ignoring base‑rate effects constitutes a fallacy and that recognizing this yields lessons for improving scientific practice.

Abstract

The replication (replicability, reproducibility) crisis in social psychology and clinical medicine arises from the fact that many apparently well-confirmed experimental results are subsequently overturned by studies that aim to replicate the original study. The culprit is widely held to be poor science: questionable research practices, failure to publish negative results, bad incentives, and even fraud. In this article I argue that the high rate of failed replications is consistent with high-quality science. We would expect this outcome if the field of science in question produces a high proportion of false hypotheses prior to testing. If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%. Failure to recognize this is to commit the fallacy of ignoring the base rate. I argue that this is a plausible diagnosis of the replication crisis and examine what lessons we thereby learn for the future conduct of science.