Modelling what users see when they look at images: a cognitive viewpoint

TLDR

User viewing and query‑matching behavior shows that image relevance can stem from object descriptions and content elements not evident in the image. The study examines how users assign query terms to retrieved images and how post‑retrieval engagement informs cognitive assessments, proposing a system that captures human interpretations to improve image retrieval beyond traditional indexing and content extraction. Affective/emotion‑based query terms are identified as an important descriptive category for image retrieval.

Abstract

Analysis of user viewing and query‐matching behavior furnishes additional evidence that the relevance of retrieved images for system users may arise from descriptions of objects and content‐based elements that are not evident or not even present in the image. This investigation looks at how users assign pre‐determined query terms to retrieved images, as well as looking at a post‐retrieval process of image engagement to user cognitive assessments of meaningful terms. Additionally, affective/emotion‐based query terms appear to be an important descriptive category for image retrieval. A system for capturing (eliciting) human interpretations derived from cognitive engagements with viewed images could further enhance the efficiency of image retrieval systems stemming from traditional indexing methods and technology‐based content extraction algorithms. An approach to such a system is posited.