Controversies Around the Role of Statistical Tests in Experimental Research

TLDR

Significance testing is widely used but has long been criticized for misinterpretation and researchers’ overconfidence, prompting calls to transform it into a more intelligent process. The article aims to clarify the logic of Fisher and Neyman‑Pearson statistical tests, review common misinterpretations, analyze underlying philosophical and psychological factors, and propose educational strategies to improve inference use. The authors conceptually dissect Fisher and Neyman‑Pearson testing frameworks, examine misinterpretations of core concepts, and analyze how philosophical and psychological issues contribute to misuse. The review concludes that most criticisms of statistical tests stem from researchers’ misuse rather than flaws in the tests themselves.

Abstract

Abstract Despite widespread use of significance testing in empirical research, its interpretation and researchers' excessive confidence in its results have been criticized for years. In this article, the logic of statistical testing in the Fisher and Neyman-Pearson approaches are described, some common misinterpretations of basic concepts behind statistical tests are reviewed, and the philosophical and psychological issues that can contribute to these misinterpretations are analyzed. Some frequent criticisms against statistical tests are revisited, with the conclusion that most of them refer not to the tests themselves but to the misuse of tests on the part of researchers. In accordance with Levin (1998a), statistical tests should be transformed into a more intelligent process that helps researchers in their work. Possible ways in which statistical education might contribute to the better understanding and application of statistical inference are suggested.