Lie for a Dime

TLDR

Online recruitment via the Internet enables large, targeted samples, but data quality hinges on participants’ honesty when eligibility is determined by self‑report. The study offers practical recommendations to exclude ineligible participants in self‑report‑based data collection. Four Amazon Mechanical Turk studies revealed that many participants misrepresent key characteristics to qualify, and that a substantial fraction of responses are impostors when recruiting rare populations.

Abstract

The Internet has enabled recruitment of large samples with specific characteristics. However, when researchers rely on participant self-report to determine eligibility, data quality depends on participant honesty. Across four studies on Amazon Mechanical Turk, we show that a substantial number of participants misrepresent theoretically relevant characteristics (e.g., demographics, product ownership) to meet eligibility criteria explicit in the studies, inferred by a previous exclusion from the study or inferred in previous experiences with similar studies. When recruiting rare populations, a large proportion of responses can be impostors. We provide recommendations about how to ensure that ineligible participants are excluded that are applicable to a wide variety of data collection efforts, which rely on self-report.