Power Analysis and Effect Size in Mixed Effects Models: A Tutorial

TLDR

Psychology faces low replication rates, with less than 40 % of studies replicated, partly because cognitive‑psychology designs involving stimulus–participant samples are not accommodated by standard power formulas. This tutorial reviews the literature on power analysis for mixed‑effects models and introduces recent software packages, illustrated with two high‑power masked‑priming studies. The authors estimate study power and assess how sample sizes could be reduced while maintaining adequate power, applying these methods to the two datasets. They recommend at least 1,600 word‑observations per condition for properly powered reaction‑time experiments, emphasize the need to report observation counts in meta‑analyses, and show that the analyses readily generalize to new data.

Abstract

In psychology, attempts to replicate published findings are less successful than expected. For properly powered studies replication rate should be around 80%, whereas in practice less than 40% of the studies selected from different areas of psychology can be replicated. Researchers in cognitive psychology are hindered in estimating the power of their studies, because the designs they use present a sample of stimulus materials to a sample of participants, a situation not covered by most power formulas. To remedy the situation, we review the literature related to the topic and introduce recent software packages, which we apply to the data of two masked priming studies with high power. We checked how we could estimate the power of each study and how much they could be reduced to remain powerful enough. On the basis of this analysis, we recommend that a properly powered reaction time experiment with repeated measures has at least 1,600 word observations per condition (e.g., 40 participants, 40 stimuli). This is considerably more than current practice. We also show that researchers must include the number of observations in meta-analyses because the effect sizes currently reported depend on the number of stimuli presented to the participants. Our analyses can easily be applied to new datasets gathered.