Dual cross-media relevance model for image annotation

TLDR

Image annotation is a key research area for image understanding and web retrieval, yet relevance‑model methods that maximize joint image‑word probability over training data suffer from a semantic gap and limited scalability. This work proposes a dual cross‑media relevance model that estimates joint image‑word probability by averaging over a predefined lexicon. DCMRM models both word‑to‑image and word‑to‑word relations, estimating them through web‑search techniques and available training data. Experiments on the Corel and a web image dataset show that DCMRM outperforms existing relevance‑model approaches.

Abstract

Image annotation has been an active research topic in recent years due to its potential impact on both image understanding and web image retrieval. Existing relevance-model-based methods perform image annotation by maximizing the joint probability of images and words, which is calculated by the expectation over training images. However, the semantic gap and the dependence on training data restrict their performance and scalability. In this paper, a dual cross-media relevance model (DCMRM) is proposed for automatic image annotation, which estimates the joint probability by the expectation over words in a pre-defined lexicon. DCMRM involves two kinds of critical relations in image annotation. One is the word-to-image relation and the other is the word-to-word relation. Both relations can be estimated by using search techniques on the web data as well as available training data. Experiments conducted on the Corel dataset and a web image dataset demonstrate the effectiveness of the proposed model.