When effect sizes disagree: The case of r and d.

TLDR

The increased use of effect sizes in single studies and meta‑analyses raises questions about statistical inference, as the choice of index can substantially affect interpretation. Because neither statistic is universally superior, researchers should explicitly consider base rates to formulate correct inferences and justify the selection of a primary effect‑size statistic. The authors illustrate the issue by comparing the correlation coefficient and the standardized mean difference (Cohen’s d or Hedges’s g), both applicable when one variable is dichotomous and the other quantitative. Although the indices are often practically interchangeable, differences in sensitivity to base rate or variance of the dichotomous variable can alter conclusions about effect magnitude depending on which statistic is used.

Abstract

The increased use of effect sizes in single studies and meta-analyses raises new questions about statistical inference. Choice of an effect-size index can have a substantial impact on the interpretation of findings. The authors demonstrate the issue by focusing on two popular effect-size measures, the correlation coefficient and the standardized mean difference (e.g., Cohen's d or Hedges's g), both of which can be used when one variable is dichotomous and the other is quantitative. Although the indices are often practically interchangeable, differences in sensitivity to the base rate or variance of the dichotomous variable can alter conclusions about the magnitude of an effect depending on which statistic is used. Because neither statistic is universally superior, researchers should explicitly consider the importance of base rates to formulate correct inferences and justify the selection of a primary effect-size statistic.