Data Collection via Online Platforms: Challenges and Recommendations for Future Research

TLDR

Online platforms such as Amazon's Mechanical Turk are increasingly used for survey and experimental data collection, yet they pose challenges in obtaining representative samples, ensuring data quality, ethically incentivizing participants, and maintaining transparency, making them a tumultuous terrain for researchers and reviewers. The article seeks to clarify these challenges by providing recommendations to both researchers and reviewers. It achieves this by presenting a series of recommendations for effectively executing and evaluating data collection via online platforms.

Abstract

Online platforms such as Amazon's Mechanical Turk (MTurk) are increasingly used by researchers to collect survey and experimental data. Yet, such platforms often represent a tumultuous terrain for both researchers and reviewers. Researchers have to navigate the complexities of obtaining representative samples from online participant cohorts, ensuring data quality, ethically incentivizing participant engagement, and maintaining transparency. Reviewers, on the other hand, have to navigate the complexities of evaluating the efficacy of such data collection and execution efforts in answering important research questions. In order to provide clarity to these issues, this article provides researchers and reviewers with a series of recommendations for effectively executing and evaluating data collection via online platforms, respectively.