Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling

TLDR

Visual storytelling generates a narrative from a sequence of photos, requiring human‑like narration beyond factual captions, yet current datasets provide only a few images per story, making it hard to bridge visual gaps. This work aims to explicitly learn to imagine a storyline that fills the visual gaps between photos. The authors train a hide‑and‑tell model that randomly omits photos during training, learns non‑local relations across the stream, and refines conventional RNNs to produce complete stories. Experiments demonstrate that the hide‑and‑tell scheme outperforms prior state‑of‑the‑art methods on automatic metrics and qualitatively interpolates storylines across missing images.

Abstract

Visual storytelling is a task of creating a short story based on photo streams. Unlike existing visual captioning, storytelling aims to contain not only factual descriptions, but also human-like narration and semantics. However, the VIST dataset consists only of a small, fixed number of photos per story. Therefore, the main challenge of visual storytelling is to fill in the visual gap between photos with narrative and imaginative story. In this paper, we propose to explicitly learn to imagine a storyline that bridges the visual gap. During training, one or more photos is randomly omitted from the input stack, and we train the network to produce a full plausible story even with missing photo(s). Furthermore, we propose for visual storytelling a hide-and-tell model, which is designed to learn non-local relations across the photo streams and to refine and improve conventional RNN-based models. In experiments, we show that our scheme of hide-and-tell, and the network design are indeed effective at storytelling, and that our model outperforms previous state-of-the-art methods in automatic metrics. Finally, we qualitatively show the learned ability to interpolate storyline over visual gaps.