Long-Term Feature Banks for Detailed Video Understanding

TLDR

Human perception relies on contextualizing current events with past information. The paper aims to extend video models to incorporate long‑term context. They introduce a long‑term feature bank that aggregates information across the full video to supplement short‑clip models. Adding the feature bank to 3D CNNs achieves state‑of‑the‑art performance on AVA, EPIC‑Kitchens, and Charades. Code is available online.

Abstract

To understand the world, we humans constantly need to relate the present to the past, and put events in context. In this paper, we enable existing video models to do the same. We propose a long-term feature bank—supportive information extracted over the entire span of a video—to augment state-of-the-art video models that otherwise would only view short clips of 2-5 seconds. Our experiments demonstrate that augmenting 3D convolutional networks with a long-term feature bank yields state-of-the-art results on three challenging video datasets: AVA, EPIC-Kitchens, and Charades. Code is available online.