2011 · 169 citations · 22 references
Social SciencesComputer ScienceMachine LearningNatural Language ProcessingPredictive AnalyticsBiasCrowd BehaviorHuman ErrorPredictive Human Resource AnalyticsReputationReputation ManagementCrowdsourcingCrowdsourcing Decision-makingElectronic SubmissionsOutcome MeasuresPredictive Scoring ModelsTask Performance AnalysisTask Performance Metrics
Detecting and correcting low quality submissions in crowdsourcing tasks is an important challenge. Prior work has primarily focused on worker outcomes or reputation, using approaches such as agreement across workers or with a gold standard to evaluate quality. We propose an alternative and complementary technique that focuses on the way workers work rather than the products they produce. Our technique captures behavioral traces from online crowd workers and uses them to predict outcome measures such quality, errors, and the likelihood of cheating. We evaluate the effectiveness of the approach across three contexts including classification, generation, and comprehension tasks. The results indicate that we can build predictive models of task performance based on behavioral traces alone, and that these models generalize to related tasks. Finally, we discuss limitations and extensions of the approach.
22
Mark Hall, Eibe Frank, Geoffrey Holmes et al. · ACM SIGKDD Explorations Newsletter · 2009 · 17.8K citations
Labeling images with a computer game
Luis von Ahn, Laura Dabbish · 2004 · 2.2K citations
Crowdsourcing user studies with Mechanical Turk
Aniket Kittur, Ed H., Bongwon Suh · 2008 · 2K citations
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text