Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

TLDR

Deep convolutional networks dominate image interpretation, but recurrent convolutional models—doubly deep in space and time—may better handle sequential visual tasks. The study proposes end‑to‑end trainable recurrent convolutional architectures and evaluates them on activity recognition, image captioning, and video description. The models integrate nonlinear state updates, back‑propagation‑trainable recurrent layers, and joint training with modern convolutional networks to learn temporal dynamics and perceptual representations. Experiments demonstrate that these recurrent convolutional models outperform separately defined or optimized state‑of‑the‑art recognition and generation models.

Abstract

Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent are effective for tasks involving sequences, visual and otherwise. We describe a class of recurrent convolutional architectures which is end-to-end trainable and suitable for large-scale visual understanding tasks, and demonstrate the value of these models for activity recognition, image captioning, and video description. In contrast to previous models which assume a fixed visual representation or perform simple temporal averaging for sequential processing, recurrent convolutional models are "doubly deep" in that they learn compositional representations in space and time. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Differentiable recurrent models are appealing in that they can directly map variable-length inputs (e.g., videos) to variable-length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent sequence models are directly connected to modern visual convolutional network models and can be jointly trained to learn temporal dynamics and convolutional perceptual representations. Our results show that such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined or optimized.