Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli.

TLDR

Researchers designing experiments where participants respond to a sample of stimuli face difficult questions about optimal design, and conventional power analysis fails because it does not treat stimuli as a random source of variation. This article introduces new power‑analysis methods for designs with crossed random factors and offers practical guidance for psychologists planning such studies. The authors evaluate five common experimental designs, detail how to estimate power under each, examine designs with multiple stimulus blocks, and provide a flexible web‑based application to compute power with default parameters. They conclude that statistical power in crossed designs plateaus at a maximum that depends on the stimulus sample size, present rules of thumb for optimal design, and supply a web tool to aid researchers.

Abstract

Researchers designing experiments in which a sample of participants responds to a sample of stimuli are faced with difficult questions about optimal study design. The conventional procedures of statistical power analysis fail to provide appropriate answers to these questions because they are based on statistical models in which stimuli are not assumed to be a source of random variation in the data, models that are inappropriate for experiments involving crossed random factors of participants and stimuli. In this article, we present new methods of power analysis for designs with crossed random factors, and we give detailed, practical guidance to psychology researchers planning experiments in which a sample of participants responds to a sample of stimuli. We extensively examine 5 commonly used experimental designs, describe how to estimate statistical power in each, and provide power analysis results based on a reasonable set of default parameter values. We then develop general conclusions and formulate rules of thumb concerning the optimal design of experiments in which a sample of participants responds to a sample of stimuli. We show that in crossed designs, statistical power typically does not approach unity as the number of participants goes to infinity but instead approaches a maximum attainable power value that is possibly small, depending on the stimulus sample. We also consider the statistical merits of designs involving multiple stimulus blocks. Finally, we provide a simple and flexible Web-based power application to aid researchers in planning studies with samples of stimuli.