Paying More Attention to Saliency

TLDR

Image captioning has recently attracted attention due to deep architectures that combine CNNs for image representation and RNNs for caption generation, while saliency prediction models that estimate human eye fixations have also advanced, yet integrating saliency into captioning remains challenging. This study proposes a captioning method that lets a generative RNN attend to different image regions during caption generation by conditioning on a saliency prediction model that distinguishes salient from contextual areas. The approach employs a recurrent neural network whose attention mechanism is guided by saliency maps, enabling the model to focus selectively on salient image parts while generating captions. Experiments on large-scale datasets demonstrate that the proposed model outperforms captioning baselines with and without saliency and surpasses state‑of‑the‑art saliency‑based captioning methods.

Abstract

Image captioning has been recently gaining a lot of attention thanks to the impressive achievements shown by deep captioning architectures, which combine Convolutional Neural Networks to extract image representations and Recurrent Neural Networks to generate the corresponding captions. At the same time, a significant research effort has been dedicated to the development of saliency prediction models, which can predict human eye fixations. Even though saliency information could be useful to condition an image captioning architecture, by providing an indication of what is salient and what is not, research is still struggling to incorporate these two techniques. In this work, we propose an image captioning approach in which a generative recurrent neural network can focus on different parts of the input image during the generation of the caption, by exploiting the conditioning given by a saliency prediction model on which parts of the image are salient and which are contextual. We show, through extensive quantitative and qualitative experiments on large-scale datasets, that our model achieves superior performance with respect to captioning baselines with and without saliency and to different state-of-the-art approaches combining saliency and captioning.