2020 · 85 citations · 32 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingMulti-task LearningNatural Language UnderstandingVisual Question AnsweringVisual SemanticsEvaluation MetricsImage CaptioningImage Feature RepresentationImage RepresentationInductive BiasMulti-task Deep LearningMultimodal Representation LearningVisual Relationship Detection
Image captioning is a multimodal problem that has attracted extensive attention in both natural language processing and computer vision. The paper proposes a novel image captioning architecture that better exploits caption semantics to improve image representation and caption generation. The architecture builds caption‑guided visual relationship graphs via weakly supervised multi‑instance learning, enriches representations with neighboring and contextual nodes, and uses multi‑task learning during generation to jointly predict words and object/predicate tags. Experiments on MSCOCO demonstrate that the framework significantly outperforms baselines, achieving state‑of‑the‑art results across multiple metrics, and the code is publicly available.
Image captioning is a multimodal problem that has drawn extensive attention in both the natural language processing and computer vision community. In this paper, we present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. The representation is then enhanced with neighbouring and contextual nodes with their textual and visual features. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences. We perform extensive experiments on the MSCOCO dataset, showing that the proposed framework significantly outperforms the baselines, resulting in the state-of-the-art performance under a wide range of evaluation metrics. The code of our paper has been made publicly available.
32
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick et al. · arXiv (Cornell University) · 2015 · 18.2K citations · Full text
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin · 2004 · 8.3K citations