2023 · 45 citations · 13 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingDeep LearningImage RetrievalPattern RecognitionRepresentation LearningAttentionAttention MechanismsCross-modal AlignmentCross-modal Representation LearningCross-modal RetrievalImage-text AlignmentMulti-modal RetrievalSemantic Alignment
Text-based person retrieval aims at searching for a pedestrian image from multiple candidates with textual descriptions. It is challenging due to uncertain cross-modal alignments caused by the large intra-class variations. To address the challenge, most existing approaches rely on various attention mechanisms and auxiliary information, yet still struggle with the uncertain cross-modal alignments arising from significant intra-class variation, leading to coarse retrieval results. To this end, we propose a novel framework termed Deep Cross-modal Evidential Learning (DCEL), which deploys evidential deep learning to consider the cross-modal alignment uncertainty. Our DCEL model comprises three components: (1) Bidirectional Evidential Learning, which models alignment uncertainty to measure and mitigate the influence of large intra-class variation; (2) Multi-level Semantic Alignment, which leverages a proposed Semantic Filtration module and image-text similarity distribution to facilitate cross-modal alignments; (3) Cross-modal Relation Learning, which reasons about latent correspondences between multi-level tokens of image and text. Finally, we integrate the advantages of the three proposed components to enhance the model to achieve reliable cross-modal alignments. Our DCEL method consistently outperforms more than ten state-of-the-art methods in supervised, weakly supervised, and domain generalization settings on three benchmarks: CUHK-PEDES, ICFG-PEDES, and RSTPReid.
13
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al. · arXiv (Cornell University) · 2021 · 5.3K citations · Full text
CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning
Huaishao Luo, Lei Ji, Ming Zhong et al. · Neurocomputing · 2022 · 677 citations
DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval
Aichun Zhu, Zijie Wang, Yifeng Li et al. · 2021 · 243 citations