IEEE Internet Computing · 2012 · 58 citations · 12 references
ComputingComputer ScienceInformation RetrievalMachine LearningNatural Language ProcessingEvaluation Of Retrieval SystemsRelevance FeedbackRelevance RankingRetrieval ModelsGround Truth Relevance JudgmentsInformation Retrieval EvaluationInformation Retrieval QualityRelevance AssessmentRelevance EstimationRelevance Feedback EvaluationRelevance JudgmentsSpam Detection In Retrieval
The performance of information retrieval (IR) systems is commonly evaluated using a test set with known relevance. Crowdsourcing is one method for learning the relevant documents to each query in the test set. However, the quality of relevance learned through crowdsourcing can be questionable, because it uses workers of unknown quality with possible spammers among them. To detect spammers, the authors' algorithm compares judgments between workers; they evaluate their approach by comparing the consistency of crowdsourced ground truth to that obtained from expert annotators and conclude that crowdsourcing can match the quality obtained from the latter.
12
Crowdsourcing user studies with Mechanical Turk
Aniket Kittur, Ed H., Bongwon Suh · 2008 · 2K citations
Quality management on Amazon Mechanical Turk
Panagiotis G. Ipeirotis, Foster Provost, Jing Wang · 2010 · 988 citations
Are your participants gaming the system?
Julie S. Downs, Mandy B. Holbrook, Steve Sheng et al. · 2010 · 427 citations