Normalized and Geometry-Aware Self-Attention Network for Image Captioning

TLDR

Self‑attention networks have proven highly effective for image captioning, yet prior work applied normalization only outside the attention mechanism; we show that internal normalization is feasible and beneficial. This study aims to enhance self‑attention for image captioning by introducing internal normalization and geometry‑aware attention modules. We design a Normalized Self‑Attention module that embeds layer‑style normalization within the attention computation, and a Geometry‑aware Self‑Attention module that incorporates relative spatial relations, then combine both into a unified captioning model. On the MS‑COCO dataset, the combined modules outperform state‑of‑the‑art baselines, and the approach generalizes to video captioning, machine translation, and visual question answering.

Abstract

Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we propose Normalized Self-Attention (NSA), a reparameterization of SA that brings the benefits of normalization inside SA. While normalization is previously only applied outside SA, we introduce a novel normalization method and demonstrate that it is both possible and beneficial to perform it on the hidden activations inside SA. Second, to compensate for the major limit of Transformer that it fails to model the geometry structure of the input objects, we propose a class of Geometry-aware Self-Attention (GSA) that extends SA to explicitly and efficiently consider the relative geometry relations between the objects in the image. To construct our image captioning model, we combine the two modules and apply it to the vanilla self-attention network. We extensively evaluate our proposals on MS-COCO image captioning dataset and superior results are achieved when comparing to state-of-the-art approaches. Further experiments on three challenging tasks, i.e. video captioning, machine translation, and visual question answering, show the generality of our methods.