Less is more

TLDR

Information retrieval systems traditionally aim to maximize the number of relevant documents returned, but this strategy can be suboptimal when users only need a limited set of relevant results. We demonstrate that attempting to return many relevant documents can actually lower the probability of retrieving any relevant documents in such scenarios.

Abstract

Traditionally, information retrieval systems aim to maximize the number of relevant documents returned to a user within some window of the top. For that goal, the probability ranking principle, which ranks documents in decreasing order of probability of relevance, is provably optimal. However, there are many scenarios in which that ranking does not optimize for the users information need. One example is when the user would be satisfied with some limited number of relevant documents, rather than needing all relevant documents. We show that in such a scenario, an attempt to return many relevant documents can actually reduce the chances of finding any relevant documents.