2023 · 22 citations · 34 references
ComputingComputer ScienceComputer VisionNatural Language ProcessingSemanticsLanguage ModelingScene UnderstandingSemantic NetworksVideo AnalysisDecoder DesignEncoder DesignHierarchical AlignmentLanguage UnderstandingMultimodal Representation LearningSemantic AlignmentVideo Representation LearningVideo Scene Understanding
Video Paragraph Grounding (VPG) is an essential yet challenging task in vision-language understanding, which aims to jointly localize multiple events from an untrimmed video with a paragraph query description. One of the critical challenges in addressing this problem is to comprehend the complex semantic relations between visual and textual modalities. Previous methods focus on modeling the contextual information between the video and text from a single-level perspective (i.e., the sentence level), ignoring rich visual-textual correspondence relations at different semantic levels, e.g., the video-word and video-paragraph correspondence. To this end, we propose a novel Hierarchical Semantic Correspondence Network (HSCNet), which explores multi-level visual-textual correspondence by learning hierarchical semantic alignment and utilizes dense supervision by grounding diverse levels of queries. Specifically, we develop a hierarchical encoder that encodes the multi-modal inputs into semantics-aligned representations at different levels. To exploit the hierarchical semantic correspondence learned in the encoder for multi-level supervision, we further design a hierarchical decoder that progressively performs finer grounding for lower-level queries conditioned on higher-level semantics. Extensive experiments demonstrate the effectiveness of HSCNet and our method significantly outstrips the state-of-the-arts on two challenging benchmarks, i.e., ActivityNet-Captions and TACoS.
34
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Learning Spatiotemporal Features with 3D Convolutional Networks
Du Tran, Lubomir Bourdev, Rob Fergus et al. · 2015 · 9.7K citations