Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).

TLDR

Bem (2011) conducted nine studies with over 1,000 participants to test whether psi—future events retroactively influencing responses—exists. The authors aim to critique Bem’s psi experiments, highlighting exploratory analysis and liberal one‑sided p‑values, and to argue that convincing skeptics requires strictly confirmatory studies with conservative statistical tests. They reanalyze Bem’s data using a default Bayesian t‑test and examine the exploratory nature of the original analysis and the use of one‑sided p‑values. The reanalysis reveals weak to nonexistent evidence for psi, indicating that Bem’s p‑values do not support precognition but instead highlight the need for changes in experimental design and data analysis.

Abstract

Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.