Novelty and diversity in information retrieval evaluation

TLDR

Evaluation measures serve as objective functions for IR systems and must accurately reflect user requirements, yet current measures poorly capture query ambiguity and document redundancy. This paper proposes an evaluation framework that systematically rewards novelty and diversity. The framework is instantiated as a cumulative‑gain‑based evaluation measure. Feasibility is shown using a TREC question‑answering test collection.

Abstract

Evaluation measures act as objective functions to be optimized by information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning IR systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents are poorly reflected by current evaluation measures. In this paper, we present a framework for evaluation that systematically rewards novelty and diversity. We develop this framework into a specific evaluation measure, based on cumulative gain. We demonstrate the feasibility of our approach using a test collection based on the TREC question answering track.