2023 · 35 citations · 44 references
ComputingComputer ScienceComputational LinguisticsComputer VisionNatural Language ProcessingDeep LearningLanguage ModelingScene UnderstandingVideo AnalysisAttentionContrastive LearningModal DecompositionMultimodal FusionMultimodal Representation LearningTextual AlignmentVideo Representation LearningVideo Scene Understanding
Spatio-temporal video grounding aims to localize the aligned visual tube corresponding to a language query. Existing techniques achieve such alignment by exploiting dense boundary and bounding box annotations, which can be prohibitively expensive. To bridge the gap, we investigate the weakly-supervised setting, where models learn from easily accessible video-language data without annotations. We identify that intra-sample spurious correlations among video-language components can be alleviated if the model captures the decomposed structures of video and language data. In this light, we propose a novel framework, namely WINNER, for hierarchical video-text understanding. WINNER first builds the language decomposition tree in a bottom-up manner, upon which the structural attention mechanism and top-down feature backtracking jointly build a multi-modal decomposition tree, permitting a hierarchical understanding of unstructured videos. The multi-modal decomposition tree serves as the basis for multi-hierarchy language-tube matching. A hierarchical contrastive learning objective is proposed to learn the multi-hierarchy correspondence and distinguishment with intra-sample and inter-sample video-text decomposition structures, achieving video-language decomposition structure alignment. Extensive experiments demonstrate the rationality of our design and its effectiveness beyond state-of-the-art weakly supervised methods, even some supervised methods.
44
Parallel Attention: A Unified Framework for Visual Object Discovery Through Dialogs and Queries
Bohan Zhuang, Qi Wu, Chunhua Shen et al. · 2018 · 150 citations
Wenqiao Zhang, Lei Zhu, James Thomas Patrick Decourcy Hallinan et al. · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · 2022 · 120 citations