2023 · 13 citations · 55 references
ComputingComputer ScienceLanguage AcquisitionLanguage DocumentationNatural Language ProcessingLanguage ModelingSequence LearningActivitiesDirect AlignmentKnowledge Base ConstructionModality AlignmentPseudo-labeling
In this paper we present an approach for localizing steps of procedural activities in narrated how-to videos. To deal with the scarcity of labeled data at scale, we source the step descriptions from a language knowledge base (wikiHow) containing instructional articles for a large variety of procedural tasks. Without any form of manual supervision, our model learns to temporally ground the steps of procedural articles in how-to videos by matching three modalities: frames, narrations, and step descriptions. Specifically, our method aligns steps to video by fusing information from two distinct pathways: i) direct alignment of step descriptions to frames, ii) indirect alignment obtained by composing steps-to-narrations with narrations-to-video correspondences. Notably, our approach performs global temporal grounding of all steps in an article at once by exploiting order information, and is trained with step pseudo-labels which are iteratively refined and aggressively filtered. In order to validate our model we introduce a new benchmark – HT-Step – obtained by manually annotating a 124-hour subset of HowTo100M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> with steps sourced from wikiHow articles. Experiments on this benchmark as well as zero-shot evaluations on CrossTask demonstrate that our multi-modality alignment yields dramatic gains over several baselines and prior works. Finally, we show that our inner module for matching narration-to-video outperforms by a large margin the state of the art on the HTM-Align narration-video alignment benchmark.
55
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
Kihyuk Sohn, David Berthelot, Chunliang Li et al. · arXiv (Cornell University) · 2020 · 2.3K citations · Full text
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Hao Tan, Mohit Bansal · 2019 · 2.3K citations · Full text
British Machine Vision Conference (BMVC)
Dima Damen, David Hogg · 2007 · 1.2K citations
VideoBERT: A Joint Model for Video and Language Representation Learning
Chen Sun, Austin Myers, Carl Vondrick et al. · 2019 · 1.1K citations