2017 · 95 citations · 27 references
ComputingComputer ScienceComputer VisionMemoryNatural Language ProcessingNarrative ComprehensionStory StructureVisual Question AnsweringMultimodal Representation LearningMultimodal Scene UnderstandingMultimodal StorytellingNarrative UnderstandingStory Understanding
We propose a novel memory network model named Read-Write Memory Network (RWMN) to perform question and answering tasks for large-scale, multimodal movie story understanding. The key focus of our RWMN model is to design the read network and the write network that consist of multiple convolutional layers, which enable memory read and write operations to have high capacity and flexibility. While existing memory-augmented network models treat each memory slot as an independent block, our use of multi-layered CNNs allows the model to read and write sequential memory cells as chunks, which is more reasonable to represent a sequential story because adjacent memory blocks often have strong correlations. For evaluation, we apply our model to all the six tasks of the MovieQA benchmark [24], and achieve the best accuracies on several tasks, especially on the visual QA task. Our model shows a potential to better understand not only the content in the story, but also more abstract information, such as relationships between characters and the reasons for their actions.
27
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Distributed Representations of Words and Phrases and their Compositionality
Tomáš Mikolov, Ilya Sutskever, Kai Chen et al. · arXiv (Cornell University) · 2013 · 18.1K citations · Full text