False-Positive Psychology

TLDR

The article aims to show that methodological flexibility inflates false‑positive findings and to propose a simple, low‑cost disclosure solution. The authors use computer simulations and two experiments to demonstrate how easily false hypotheses can be reported as significant, and outline six author requirements and four reviewer guidelines to curb this. They find that flexibility in data collection, analysis, and reporting dramatically raises false‑positive rates, making researchers more likely to falsely claim an effect than to correctly reject it.

Abstract

In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.