NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning

TLDR

Video learning is a growing computer vision task, and because videos contain millions of frames, methods that avoid frame‑level annotation are especially valuable. The authors propose a Viterbi‑based loss algorithm enabling online, incremental learning from weakly annotated video data. The method uses a Viterbi‑based loss to train models incrementally on weakly labeled videos. Explicit context and length modeling within the framework yields large gains, achieving up to 10 % improvement on action segmentation benchmarks over state‑of‑the‑art.

Abstract

Video learning is an important task in computer vision and has experienced increasing interest over the recent years. Since even a small amount of videos easily comprises several million frames, methods that do not rely on a frame-level annotation are of special importance. In this work, we propose a novel learning algorithm with a Viterbi-based loss that allows for online and incremental learning of weakly annotated video data. We moreover show that explicit context and length modeling leads to huge improvements in video segmentation and labeling tasks and include these models into our framework. On several action segmentation benchmarks, we obtain an improvement of up to 10% compared to current state-of-the-art methods.