Online panels in social science research: Expanding sampling methods beyond Mechanical Turk

TLDR

Amazon Mechanical Turk is a popular platform for recruiting behavioral science participants, offering advantages over traditional student pools but suffering from a small, overused sample and underrepresentation of key demographic groups. This study investigates whether online research panels can overcome these limitations. We compared MTurk and Prime Panels on sample composition, data quality (effect sizes, internal reliability, attention checks), and participant naivete. Prime Panels yielded a more diverse sample, lower exposure to classic protocols, and larger effect sizes after excluding participants who failed a screening task, indicating panels provide unique opportunities with notable trade-offs.

Abstract

Amazon Mechanical Turk (MTurk) is widely used by behavioral scientists to recruit research participants. MTurk offers advantages over traditional student subject pools, but it also has important limitations. In particular, the MTurk population is small and potentially overused, and some groups of interest to behavioral scientists are underrepresented and difficult to recruit. Here we examined whether online research panels can avoid these limitations. Specifically, we compared sample composition, data quality (measured by effect sizes, internal reliability, and attention checks), and the non-naivete of participants recruited from MTurk and Prime Panels-an aggregate of online research panels. Prime Panels participants were more diverse in age, family composition, religiosity, education, and political attitudes. Prime Panels participants also reported less exposure to classic protocols and produced larger effect sizes, but only after screening out several participants who failed a screening task. We conclude that online research panels offer a unique opportunity for research, yet one with some important trade-offs.