Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs

TLDR

Action recognition has traditionally treated actions as monolithic events, yet cognitive science suggests that people encode activities hierarchically, a perspective rarely explored in computer vision. We propose Action Genome, a representation that decomposes actions into spatio‑temporal scene graphs. Action Genome captures dynamic changes between objects and their pairwise relationships during an action and is constructed from 10 K videos annotated with 0.4 M objects and 1.7 M visual relationships. Applying Action Genome to action recognition improves performance on the Charades dataset and enables few‑shot recognition with 42.7 % mAP from only ten examples, while also providing a benchmark for spatio‑temporal scene graph prediction.

Abstract

Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However, in Computer Vision, few explorations on representations that encode event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.