Statistical Significance and Replicability

Abstract

In spite of arguments to the contrary, psychologists, it is shown here, believe statistical significance (SS) signifies that a finding will replicate. The most visible argument that SS is not an index of replicability, one that is based in notions of Bayesian statistical inference, is considered and shown to be flawed. Two different arguments are presented that demonstrate the irrelevance of SS to replicability: (a) SS may not be taken as a sign of the truth of the research hypothesis; and (b) statistical significance tests do not generate verifiable predictions of replication attempts. Direct tests of replicability and effect-size measures of replicability are shown to have comparable problems. A solution to the replicability problem is proposed for atheoretical research that replaces `once-and-for-all' tests of replicability with the requirement that the treatment effect be demonstrable (a) in the individual, (b) on a continuing basis, and (c) in a way that is clearly discernible. The question of the role of significance testing in research is answered by denying a role to such testing.