Integrating understandability in the evaluation of consumer health search engines

Abstract

In this paper we propose a method that integrates the no-tion of understandability, as a factor of document relevance, into the evaluation of information retrieval systems for con-sumer health search. We consider the gain-discount eval-uation framework (RBP, nDCG, ERR) and propose two understandability-based variants (uRBP) of rank biased pre-cision, characterised by an estimation of understandability based on document readability and by different models of how readability influences user understanding of document content. The proposed uRBP measures are empirically con-trasted to RBP by comparing system rankings obtained with each measure. The findings suggest that considering under-standability along with topicality in the evaluation of in-formation retrieval systems lead to different claims about systems effectiveness than considering topicality alone.