Regularizing RNNs for Caption Generation by Reconstructing the Past with the Present

TLDR

Caption generation using encoder‑decoder models has been widely explored in image and code captioning. The authors propose the Auto‑Reconstructor Network (ARNet) to enhance encoder‑decoder caption generation. ARNet reconstructs the previous hidden state from the current one, acting as an input‑dependent transition operator that embeds more information from the past to regularize RNN dynamics. Experiments show ARNet improves performance over existing encoder‑decoder models on image and source code captioning, reduces training‑inference discrepancy, and better regularizes RNNs on permuted sequential MNIST, especially for long‑term dependencies. Code is available at https://github.com/chenxinpeng/ARNet.

Abstract

Recently, caption generation with an encoder-decoder framework has been extensively studied and applied in different domains, such as image captioning, code captioning, and so on. In this paper, we propose a novel architecture, namely Auto-Reconstructor Network (ARNet), which, coupling with the conventional encoder-decoder framework, works in an end-to-end fashion to generate captions. ARNet aims at reconstructing the previous hidden state with the present one, besides behaving as the input-dependent transition operator. Therefore, ARNet encourages the current hidden state to embed more information from the previous one, which can help regularize the transition dynamics of recurrent neural networks (RNNs). Extensive experimental results show that our proposed ARNet boosts the performance over the existing encoder-decoder models on both image captioning and source code captioning tasks. Additionally, ARNet remarkably reduces the discrepancy between training and inference processes for caption generation. Furthermore, the performance on permuted sequential MNIST demonstrates that ARNet can effectively regularize RNN, especially on modeling long-term dependencies. Our code is available at: https://github.com/chenxinpeng/ARNet.