Statistical significance in psychological research.

TLDR

In psychological research, theories typically predict only the direction of effects, and because the null hypothesis is never strictly true, statistical significance—especially from a single test—provides limited confirmation, underscoring the need for multiple corroborations and constructive replications. The study tests Sapolsky’s hypothesis that psychiatric patients holding an unconscious birth‑theory belief will exhibit eating disorders and Rorschach responses involving frogs.

Abstract

Most theories in the areas of personality, clinical, and social psychology predict no more than the direction of a correlation, group difference, or treatment effect. Since the null hypothesis is never strictly true, such predictions have about a SO-SO chance of being confirmed by experiment when the theory in question is false, since the statistical significance of the result is a function of the sample size. Confirmation of a single directional prediction should usually add little to one's confidence in the theory being tested. Most theories should be tested by multiple corroboration and most empirical generalizations by constructive replication. Statistical significance is perhaps the least important attribute of a good experiment; it is never a sufficient condition for claiming that a theory has been usefully corroborated, that a meaningful empirical fact has been established, or that an experimental report ought to be published. In a recent journal article Sapolsky (1964) developed the following substantive theory: Some psychiatric patients entertain an unconscious belief in the theory of birth which involves the notions of oral impregnation and anal parturition. Such patients should be inclined to manifest eating disorders: compulsive eating in the case of those who wish to get pregnant and anorexia in those who do not. Such patients should also be inclined to see cloacal animals, such as frogs, on the Rorschach. This reasoning led Sapolsky to predict that Rorschach frog responders show