Proceedings of the AAAI Conference on Artificial Intelligence · 2017 · 90 citations · 26 references
ComputingComputer ScienceComputer VisionNatural Language ProcessingVisual AttentionFeature DescriptionLanguage ModelingNatural Language UnderstandingScene UnderstandingVisual Question AnsweringAttentionAttention GuidanceCaptioningImage CaptioningMetricsScene Description LearningVisual Attention GuidanceVisual Scene Description
Visual attention is crucial for image understanding and has proven effective in generating natural language descriptions, while recent studies indicate that associated language can steer visual attention during cognition. The study introduces a text‑guided attention model for image captioning that learns to drive visual attention using associated captions. The model employs an exemplar‑based learning approach that retrieves associated captions from training data for each image and uses them to learn attention over visual features. The model achieves state‑of‑the‑art performance on the MS‑COCO Captioning benchmark and improves scene description by accurately distinguishing small or confusable objects.
Visual attention plays an important role to understand images and demonstrates its effectiveness in generating natural language descriptions of images. On the other hand, recent studies show that language associated with an image can steer visual attention in the scene during our cognitive process. Inspired by this, we introduce a text-guided attention model for image captioning, which learns to drive visual attention using associated captions. For this model, we propose an exemplar-based learning approach that retrieves from training data associated captions with each image, and use them to learn attention on visual features. Our attention model enables to describe a detailed state of scenes by distinguishing small or confusable objects effectively. We validate our model on MS-COCO Captioning benchmark and achieve the state-of-the-art performance in standard metrics.
26
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia et al. · 2015 · 46.8K citations
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Jun‐Young Chung, Çağlar Gülçehre, Kyunghyun Cho et al. · arXiv (Cornell University) · 2014 · 10.8K citations · Full text
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Kelvin Xu, Jimmy Ba, Ryan Kiros et al. · arXiv (Cornell University) · 2015 · 7.5K citations · Full text