Weakly Supervised Action Localization by Sparse Temporal Pooling Network

TLDR

The paper proposes a weakly supervised temporal action localization algorithm for untrimmed videos using convolutional neural networks. The method learns from video‑level labels, identifies a sparse set of key segments with an attention module, fuses them through adaptive temporal pooling, and uses a dual‑term loss to enforce classification accuracy and sparsity, then extracts temporal proposals via class activations and attentions. The algorithm achieves state‑of‑the‑art performance on THUMOS14 and strong results on ActivityNet1.3 under weak supervision.

Abstract

We propose a weakly supervised temporal action localization algorithm on untrimmed videos using convolutional neural networks. Our algorithm learns from video-level class labels and predicts temporal intervals of human actions with no requirement of temporal localization annotations. We design our network to identify a sparse subset of key segments associated with target actions in a video using an attention module and fuse the key segments through adaptive temporal pooling. Our loss function is comprised of two terms that minimize the video-level action classification error and enforce the sparsity of the segment selection. At inference time, we extract and score temporal proposals using temporal class activations and class-agnostic attentions to estimate the time intervals that correspond to target actions. The proposed algorithm attains state-of-the-art results on the THUMOS14 dataset and outstanding performance on ActivityNet1.3 even with its weak supervision.