Characterizing relevance with eye-tracking measures

Abstract

Relevance, a fundamental concept in information search and retrieval, is 80-years old [4]. The recent decades have been ripe with work that brought a much better understanding of this rich concept. Yet, we still don't know which cognitive and affective processes are involved in relevance judgments. Empirical work that tackles these questions is scarce. This paper aims to contribute toward better understanding of cognitive processing of text documents at different degrees of relevance. Our approach takes advantage of a direct relationship between eye movement patterns, pupil size and cognitive processes, such as mental effort and attention. We examine gaze-based metrics in relation to individual word processing and reading text documents in the context of a constricted information search tasks. The findings indicate that text document processing depends on document relevance and on the user-perceived relevance. Statistical analyses show that relevant documents tended to be continuously read, while irrelevant documents tended to be scanned. Most eye-tracking-based measures indicate cognitive effort to be highest for partially relevant documents and lowest for irrelevant documents. However, pupil dilation indicates cognitive effort to be higher for relevant than partially relevant documents. Classification of selected eye-tracking measures show that an accuracy of 70-75% can be achieved for predicting binary relevance. These results show a promise for implicit relevance feedback.