But What Do the Numbers Really Tell Us?: Arbitrary Metrics and Effect Size Reporting in Sport Psychology Research

Abstract

Many of the measurements used in sport psychology research are arbitrary metrics, and researchers often cannot make the jump from scores on paper-and-pencil tests to what those scores actually mean in terms of real-world behaviors. Effect sizes for behavioral data are often interpretable, but the meaning of a small, medium, or large effect for an arbitrary metric is elusive. We reviewed all the issues in the 2005 volumes of the Journal of Sport & Exercise Psychology, The Sport Psychologist, and the Journal of Applied Sport Psychology to determine whether the arbitrary metrics used in sport psychology research were interpreted, or calibrated, against real-world variables. Of the 54 studies that used quantitative methods, 25 reported only paper-and-pencil arbitrary metrics with no connections to behavior or other real-world variables. Also, 44 of the 54 studies reported effect sizes, but only 7 studies, using both arbitrary and behavioral metrics, had calculated effect indicators and interpreted them in terms of real-world meaning.