Deep Supervised Cross-Modal Retrieval

TLDR

Cross‑modal retrieval enables flexible retrieval across different modalities by measuring content similarity between heterogeneous data types. This paper introduces Deep Supervised Cross‑Modal Retrieval (DSCMR) to learn a common representation for cross‑modal comparison. DSCMR learns a shared latent space by jointly minimizing discrimination loss in label and representation spaces, modality‑invariance loss, and applying weight sharing to reduce cross‑modal discrepancy. Experiments on four benchmark datasets show DSCMR outperforms state‑of‑the‑art methods in cross‑modal retrieval.

Abstract

Cross-modal retrieval aims to enable flexible retrieval across different modalities. The core of cross-modal retrieval is how to measure the content similarity between different types of data. In this paper, we present a novel cross-modal retrieval method, called Deep Supervised Cross-modal Retrieval (DSCMR). It aims to find a common representation space, in which the samples from different modalities can be compared directly. Specifically, DSCMR minimises the discrimination loss in both the label space and the common representation space to supervise the model learning discriminative features. Furthermore, it simultaneously minimises the modality invariance loss and uses a weight sharing strategy to eliminate the cross-modal discrepancy of multimedia data in the common representation space to learn modality-invariant features. Comprehensive experimental results on four widely-used benchmark datasets demonstrate that the proposed method is effective in cross-modal learning and significantly outperforms the state-of-the-art cross-modal retrieval methods.