2008 · 625 citations · 20 references
ComputingComputer ScienceData EthicsMachine LearningNatural Language ProcessingAnnotationInformation ExtractionAnnotation SharingAnnotation ToolsAutomated AnnotationAutomatic AnnotationCrowdsourced Data CollectionData AnnotationUser AnnotationUtility Data Management
The study demonstrates how to outsource data annotation to Amazon Mechanical Turk, presents results across several annotation problems, and outlines strategies for task specification and pricing. The authors employ Amazon Mechanical Turk to outsource annotation tasks, using strategies that ensure tasks are well specified and properly priced. Outsourcing to MTurk produces large, inexpensive, high‑quality, and rapid annotations, with quality that can be checked and controlled across multiple problems.
We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.
20
Histograms of Oriented Gradients for Human Detection
Navneet Dalal, Bill Triggs · 2005 · 31.9K citations · Full text
LabelMe: A Database and Web-Based Tool for Image Annotation
Bryan Russell, Antonio Torralba, Kevin Murphy et al. · International Journal of Computer Vision · 2007 · 4.2K citations · Full text