The Demographic and Political Composition of Mechanical Turk Samples

TLDR

Online convenience samples from Amazon’s Mechanical Turk are increasingly used in survey research, yet the full scope of their social and political differences compared to population-based samples remains unclear. This study examines how MTurk samples differ from population samples and explores the underlying nature of those differences. The authors replicated items from the 2012 American National Election Studies Time Series Survey in a survey administered to MTurk respondents. MTurk respondents differ significantly from 2012 ANES Web respondents, but most differences diminish when controlling for measurable demographic and political variables, indicating that MTurk data can be used for research when such variables are measured and accounted for.

Abstract

One of the most notable recent developments in survey research is the increased usage of online convenience samples drawn from Amazon’s Mechanical Turk (MTurk). While scholars have noted various social and political differences (e.g., age, partisanship) between MTurk and population-based samples, the breadth and depth of these variations remain unclear. We investigate the extent to which MTurk samples differ from population samples, and the underlying nature of these differences. We do so by replicating items from the population-based American National Election Studies (ANES) 2012 Time Series Study in a survey administered to a sample of MTurk respondents. With few exceptions, we not only find that MTurk respondents differ significantly from respondents completing the 2012 ANES via the Web but also that most differences are reduced considerably when controlling for easily measurable sample features. Thus, MTurk respondents do not appear to differ fundamentally from population-based respondents in unmeasurable ways. This suggests that MTurk data can be used to advance research programs, particularly if researchers measure and account for a range of political and demographic variables as needed.