The Extent and Consequences of P-Hacking in Science

TLDR

Focusing on novel, confirmatory, and statistically significant results biases the scientific literature, with p‑hacking—selecting or collecting data until nonsignificant results become significant—being a key contributor. The study aims to demonstrate that p‑hacking is widespread throughout science. The authors employ text‑mining techniques to identify evidence of p‑hacking across scientific literature. They show that p‑hacking is common but its impact is weak compared to true effect sizes, suggesting it does not drastically alter scientific consensuses derived from meta‑analyses.

Abstract

A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as "p-hacking," occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.