Are samples drawn from Mechanical Turk valid for research on political ideology?

Scott Clifford, Ryan Jewell, Philip Waggoner

Research & Politics · 2015 · 704 citations · 44 references

DOIFull text

Open access

Concepts

TL;DR

Amazon’s Mechanical Turk is widely used for recruiting research participants, yet its suitability for studying political ideology remains uncertain because it is unclear whether MTurk liberals and conservatives share the same psychological dispositions as those in the general population. The study evaluates whether MTurk samples are valid for political ideology research by comparing them to two national benchmark samples, one online and one face‑to‑face. The authors compare personality and value‑based motivations of liberals and conservatives across a large MTurk sample and the two national samples. The three samples produced virtually identical results, showing that MTurk liberals and conservatives mirror the psychological divisions of the mass public—though MTurk liberals are slightly more liberal—confirming MTurk’s validity for political ideology research.

Abstract

Amazon’s Mechanical Turk (MTurk) is an increasingly popular tool for the recruitment of research subjects. While there has been much focus on the demographic differences between MTurk samples and the national public, we know little about whether liberals and conservatives recruited from MTurk share the same psychological dispositions as their counterparts in the mass public. In the absence of such evidence, some have argued that the selection process involved in joining MTurk invalidates the subject pool for studying questions central to political science. In this paper, we evaluate this claim by comparing a large MTurk sample to two benchmark national samples – one conducted online and one conducted face-to-face. We examine the personality and value-based motivations of political ideology across the three samples. All three samples produce substantively identical results with only minor variation in effect sizes. In short, liberals and conservatives in our MTurk sample closely mirror the psychological divisions of liberals and conservatives in the mass public, though MTurk liberals hold more characteristically liberal values and attitudes than liberals from representative samples. Overall, our results suggest that MTurk is a valid recruitment tool for psychological research on political ideology.

References

44