Cheap and fast---but is it good?

TLDR

Human linguistic annotation is essential for many NLP tasks but is expensive and time‑consuming. The study investigates using Amazon Mechanical Turk as a cheaper, faster way to collect annotations and proposes a bias‑correction technique that improves quality on two tasks. The authors employed MTurk to gather annotations for five tasks—affect recognition, word similarity, textual entailment, event temporal ordering, and word sense disambiguation—and applied a bias‑correction method to enhance data quality. Across all five tasks, MTurk non‑expert annotations showed high agreement with expert gold standards, proved effective for training affect‑recognition models, and, with bias correction, enabled large‑scale labeling at a fraction of the usual cost.

Abstract

Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.