2022 · 34 citations · 26 references
ComputingComputer ScienceInformation RetrievalMachine LearningNatural Language ProcessingHash FunctionsMultimedia RetrievalBenchmark DatasetsCross-modal RetrievalHamming DistanceHashingMultimodal RetrievalMultimodal Retrieval ModelsRetrievalVideo Retrieval
With the ever-increasing multimedia data on the Web, cross-modal video-text retrieval has received a lot of attention in recent years. Deep cross-modal hashing approaches utilize the Hamming space for achieving fast retrieval. However, most existing algorithms have difficulties in seeking or constructing a well-defined joint semantic space. In this paper, an unsupervised deep cross-modal video-text hashing approach (CLIP4Hashing) is proposed, which mitigates the difficulties in bridging between different modalities in the Hamming space through building a single hashing net by employing the pre-trained CLIP model. The approach is enhanced by two novel techniques, the dynamic weighting strategy and the design of the min-max hashing layer, which are found to be the main sources of the performance gain. Compared with conventional deep cross-modal hashing algorithms, CLIP4Hashing does not require data-specific hyper-parameters. With evaluation using three challenging video-text benchmark datasets, we demonstrate that CLIP4Hashing is able to significantly outperform existing state-of-the-art hashing algorithms. Additionally, with larger bit sizes (e.g., 2048 bits), CLIP4Hashing can even deliver competitive performance compared with the results based on non-hashing features.
26
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al. · arXiv (Cornell University) · 2021 · 5.3K citations · Full text
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
A new approach to cross-modal multimedia retrieval
Nikhil Rasiwasia, José Costa Pereira, Emanuele Coviello et al. · 2010 · 1.4K citations
Deep Supervised Hashing for Fast Image Retrieval
Haomiao Liu, Ruiping Wang, Shiguang Shan et al. · 2016 · 929 citations
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling
Jie Lei, Linjie Li, Luowei Zhou et al. · 2021 · 552 citations