Mining Similarity Relationships for Unsupervised Cross-Modal Hashing

Abstract

Previous methods have made promising progresses, but there are still some limitations in narrowing the gap between modalities and exploring and preserving intrinsic multimodal semantics. Furthermore, there has been a failure to effectively incorporate the generation of hash codes during the training process to correct poorly trained instance pairs. To address these issues, we propose a new unsupervised hash learning framework Mining Similarity Relationships Hashing(MSRH). First, we construct a multimodal fusion similarity map that nonlinearly combines intra-modal and inter-modal similarity maps. Furthermore, we build a feature cross reconstruction module to reduce the gap between modalities. This module fully explores the semantic correlation between intra-modal and inter-modal representations and generates multi-modal representations with complementary relationships. Finally, we propose a multi-modal fusion graph update module to update poorly trained instance pairs to further improve retrieval performance. Extensive experiments on two widely used datasets demonstrate the effectiveness and superiority of this method.