2009 · 839 citations · 25 references
ComputingComputer ScienceInformation RetrievalText AnalyticsWeb AnalyticsEvaluation Of Retrieval SystemsRelevance FeedbackRelevance RankingRetrieval ModelsSearch Log AnalysisSearch SystemsBinary RelevanceDiscounted Cumulative GainExpected Reciprocal RankGraded RelevanceInformation Retrieval EvaluationMetricsReciprocal RankSearch Engine Query Logs
While numerous metrics for information retrieval are available in the case of binary relevance, there is only one commonly used metric for graded relevance, namely the Discounted Cumulative Gain (DCG). A drawback of DCG is its additive nature and the underlying independence assumption: a document in a given position has always the same gain and discount independently of the documents shown above it. Inspired by the "cascade" user model, we present a new editorial metric for graded relevance which overcomes this difficulty and implicitly discounts documents which are shown below very relevant documents. More precisely, this new metric is defined as the expected reciprocal length of time that the user will take to find a relevant document. This can be seen as an extension of the classical reciprocal rank to the graded relevance case and we call this metric Expected Reciprocal Rank (ERR). We conduct an extensive evaluation on the query logs of a commercial search engine and show that ERR correlates better with clicks metrics than other editorial metrics.
25
Learning to rank using gradient descent
Chris Burges, Tal Shaked, Erin Renshaw et al. · 2005 · 2.8K citations
Matthew Richardson, Ewa Dominowska, Robert Ragno · 2007 · 1.1K citations
Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson et al. · 2009 · 1K citations
An experimental comparison of click position-bias models
Nick Craswell, Onno Zoeter, Michael Taylor et al. · 2008 · 955 citations