Crowdsourcing and Its Applications in Computer Vision

Abstract

Crowdsourcing has emerged in the past decade as a popular model for online distributed problem solving and production. The creation of Amazon Mechanical Turk (MTurk) as a “micro-task ” marketplace has facilitated this growth by connecting willing workers with available tasks. Within computer vision, MTurk has proven especially useful in large-scale image label collection, as many computer vision algorithms require substantial amounts of training data. In this survey, we discuss different types of worker incentives, various considerations for MTurk task design, methods for annotation quality analysis, and cost-effective ways of obtaining labels in a selective manner. We present several examples of how MTurk is being utilized in the computer vision community. Finally, we discuss the implications that MTurk usage will have on future computer vision research. 1