Stimulus Sampling and Social Psychological Experimentation

TLDR

Social psychological experiments often fail to sample stimuli, a flaw that can undermine both external and construct validity. The authors identify situations where stimulus sampling is not obvious and highlight cognitive biases that cause researchers to overlook its necessity. Survey data from undergraduate, graduate, and faculty participants revealed little concern for stimulus sampling unless the issue was explicitly emphasized, and the authors discuss its statistical consequences for power, effect size, and analysis.

Abstract

The authors discuss the problem with failing to sample stimuli in social psychological experimentation. Although commonly construed as an issue for external validity, the authors emphasize how failure to sample stimuli also can threaten construct validity. They note some circumstances where the need for stimulus sampling is less obvious and more obvious, and they discuss some well-known cognitive biases that can contribute to the failure of researchers to see the need for stimulus sampling. Data are presented from undergraduate students (N = 106), graduate students (N = 72), and psychology faculty (N = 48) showing insensitivity to the need for stimulus sampling except when the problem is made rather obvious. Finally, some of the statistical implications of stimulus sampling with particular concern for power, effect size estimates, and data analysis strategies are noted.