Neural Information Processing Systems · 2015 · 88 citations · 25 references
ComputingComputer ScienceInformation RetrievalMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingMultimedia RetrievalMultimodal InteractionNatural Language UnderstandingConvolutional Neural NetworksImage-text RetrievalLocal CoherenceMultimodal Representation LearningMultimodal Retrieval ModelsNatural Language InterfacesNeural Text RepresentationRecurrent Neural Networks
We propose an approach for retrieving a sequence of natural sentences for an image stream. Since general users often take a series of pictures on their special moments, it would better take into consideration of the whole image stream to produce natural language descriptions. While almost all previous studies have dealt with the relation between a single image and a single natural sentence, our work extends both input and output dimension to a sequence of images and a sequence of sentences. To this end, we design a multimodal architecture called coherence recurrent convolutional network (CRCN), which consists of convolutional neural networks, bidirectional recurrent neural networks, and an entity-based local coherence model. Our approach directly learns from vast user-generated resource of blog posts as text-image parallel training data. We demonstrate that our approach outperforms other state-of-the-art candidate methods, using both quantitative measures (e.g. BLEU and top-K recall) and user studies via Amazon Mechanical Turk.
25
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2015 · 18.8K citations
The Stanford CoreNLP Natural Language Processing Toolkit
Christopher D. Manning, Mihai Surdeanu, John Bauer et al. · 2014 · 7.2K citations · Full text