2013 · 17 citations · 7 references
ComputingComputer ScienceInformation RetrievalMulti-criteria Decision MakingText AnalyticsEvaluation Of Retrieval SystemsPreference ElicitationRanking ProblemsRelevance FeedbackRelevance RankingRetrieval ModelsDocument RelevanceDocument Relevance ModelingPairwise Preference ElicitationPairwise Preference ModelingRanking MetricsRelevance AssessmentRelevance JudgmentsRetrieval Systems
High quality relevance judgments are essential for the evaluation of information retrieval systems. Traditional methods of collecting relevance judgments are based on collecting binary or graded nominal judgments, but such judgments are limited by factors such as inter-assessor disagreement and the arbitrariness of grades. Previous research has shown that it is easier for assessors to make pairwise preference judgments. However, unless the preferences collected are largely transitive, it is not clear how to combine them in order to obtain document relevance scores. Another difficulty is that the number of pairs that need to be assessed is quadratic in the number of documents. In this work, we consider the problem of inferring document relevance scores from pairwise preference judgments by analogy to tournaments using the Elo rating system. We show how to combine a linear number of pairwise preference judgments from multiple assessors to compute relevance scores for every document.
7
Optimizing search engines using clickthrough data
Thorsten Joachims · 2002 · 3.9K citations
Quality management on Amazon Mechanical Turk
Panagiotis G. Ipeirotis, Foster Provost, Jing Wang · 2010 · 988 citations
Pairwise ranking aggregation in a crowdsourced setting
Xi Chen, Paul N. Bennett, Kevyn Collins‐Thompson et al. · 2013 · 307 citations