TALL: Temporal Activity Localization via Language Query

TLDR

Temporal action localization in untrimmed videos is challenging because existing methods rely on predefined action lists and sliding‑window classifiers, which struggle with the diverse actors, actions, and objects found in real‑world footage. This work aims to localize activities using natural language queries instead of fixed action categories. We introduce the Cross‑modal Temporal Regression Localizer (CTRL), a model that jointly encodes text queries and video clips to produce alignment scores and boundary regressions, and we create the Charades‑STA dataset with sentence‑level temporal annotations for evaluation. CTRL achieves significant improvements over prior methods on both the TaCoS and Charades‑STA datasets.

Abstract

This paper focuses on temporal localization of actions in untrimmed videos. Existing methods typically train classifiers for a pre-defined list of actions and apply them in a sliding window fashion. However, activities in the wild consist of a wide combination of actors, actions and objects; it is difficult to design a proper activity list that meets users' needs. We propose to localize activities by natural language queries. Temporal Activity Localization via Language (TALL) is challenging as it requires: (1) suitable design of text and video representations to allow cross-modal matching of actions and language queries; (2) ability to locate actions accurately given features from sliding windows of limited granularity. We propose a novel Cross-modal Temporal Regression Localizer (CTRL) to jointly model text query and video clips, output alignment scores and action boundary regression results for candidate clips. Lor evaluation, we adopt TaCoS dataset, and build a new dataset for this task on top of Charades by adding sentence temporal annotations, called Charades-STA. We also build complex sentence queries in Charades-STA for test. Experimental results show that CTRL outperforms previous methods significantly on both datasets.