Sequence to Sequence -- Video to Text

TLDR

Real‑world videos have complex dynamics, so open‑domain video description methods must be sensitive to temporal structure and support variable‑length input and output sequences. The study proposes a novel end‑to‑end sequence‑to‑sequence model to generate captions for videos. The model uses LSTM recurrent neural networks trained on video‑sentence pairs to map variable‑length frame sequences to word sequences, and its variants are evaluated on YouTube videos and the M‑VAD and MPII‑MD movie description datasets. The model learns the temporal structure of video frames and the language model of generated sentences.

Abstract

Real-world videos often have complex dynamics, methods for generating open-domain video descriptions should be sensitive to temporal structure and allow both input (sequence of frames) and output (sequence of words) of variable length. To approach this problem we propose a novel end-to-end sequence-to-sequence model to generate captions for videos. For this we exploit recurrent neural networks, specifically LSTMs, which have demonstrated state-of-the-art performance in image caption generation. Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip. Our model naturally is able to learn the temporal structure of the sequence of frames as well as the sequence model of the generated sentences, i.e. a language model. We evaluate several variants of our model that exploit different visual features on a standard set of YouTube videos and two movie description datasets (M-VAD and MPII-MD).