2023 · 21 citations · 52 references
ComputingComputer ScienceComputer VisionKnowledge RepresentationMachine LearningAction RecognitionAction RepresentationDeep LearningLearning ActivitiesSequence LearningVideo AnalysisActivitiesProbabilistic Deep LearningSequence Representation LearningTemporal Ordering In NarrativeVideo ClassificationVideo Representation LearningZero-shot Action RecognitionZero-shot Classification
The abundance of instructional videos and their narrations over the Internet offers an exciting avenue for understanding procedural activities. In this work, we propose to learn video representation that encodes both action steps and their temporal ordering, based on a large-scale dataset of web instructional videos and their narrations, without using human annotations. Our method jointly learns a video representation to encode individual step concepts, and a deep probabilistic model to capture both temporal dependencies and immense individual variations in the step ordering. We empirically demonstrate that learning temporal ordering not only enables new capabilities for procedure reasoning, but also reinforces the recognition of individual steps. Our model significantly advances the state-of-the-art results on step classification (+2.8%/+3.3% on COIN / EPIC-Kitchens) and step forecasting (+7.4% on COIN). Moreover, our model attains promising results in zero-shot inference for step classification and fore-casting, as well as in predicting diverse and plausible steps for incomplete procedures. Our code is available at https://github.com/facebookresearch/ProcedureVRL.
52
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter · arXiv (Cornell University) · 2017 · 9.1K citations · Full text
Denoising Diffusion Probabilistic Models
arXiv (Cornell University) · 2020 · 5.6K citations · Full text