2014 · 13 citations · 20 references
ComputingComputer ScienceInformation SystemsHuman-computer InteractionInformation RetrievalUser Experience DesignEvaluation Of Retrieval SystemsInformation SearchRetrieval ModelsSearch Log AnalysisSearch SystemsWeb SearchImplicit FeedbackOnline SearchSearch Engine EvaluationSearch Engine User BehaviorSearch Performance MetricsSearch Quality EvaluationSearch Satisfaction MeasurementSearch Usability Evaluation
Online search evaluation metrics are typically derived based on implicit feedback from the users. For instance, computing the number of page clicks, number of queries, or dwell time on a search result. In a recent paper, Dupret and Lalmas introduced a new metric called absence time, which uses the time interval between successive sessions of users to measure their satisfaction with the system. They evaluated this metric on a version of Yahoo! Answers. In this paper, we investigate the effectiveness of absence time in evaluating new features in a web search engine, such as new ranking algorithm or a new user interface. We measured the variation of absence time to the effects of 21 experiments performed on a search engine. Our findings show that the outcomes of absence time agreed with the judgement of human experts performing a thorough analysis of a wide range of online and offline metrics in 14 out of these 21 cases.
20
Accurately interpreting clickthrough data as implicit feedback
Thorsten Joachims, Laura Granka, Bing Pan et al. · 2005 · 1.4K citations
Matthew Richardson, Ewa Dominowska, Robert Ragno · 2007 · 1.1K citations