Utility data annotation with Amazon Mechanical Turk

TLDR

The study demonstrates how to outsource data annotation to Amazon Mechanical Turk, presents results across several annotation problems, and outlines strategies for task specification and pricing. The authors employ Amazon Mechanical Turk to outsource annotation tasks, using strategies that ensure tasks are well specified and properly priced. Outsourcing to MTurk produces large, inexpensive, high‑quality, and rapid annotations, with quality that can be checked and controlled across multiple problems.

Abstract

We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.