arXiv (Cornell University) · 2018 · 110 citations · 0 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingDeep LearningGraph-based LearningNatural Language UnderstandingQuestion AnsweringRepresentation LearningVisual Question AnsweringBaseline ArchitectureExplainable Visual Question AnsweringGraph Data StructuresGraph Representation LearningGraph-based Reasoning
Visual Question answering is a challenging problem requiring a combination of\nconcepts from Computer Vision and Natural Language Processing. Most existing\napproaches use a two streams strategy, computing image and question features\nthat are consequently merged using a variety of techniques. Nonetheless, very\nfew rely on higher level image representations, which can capture semantic and\nspatial relationships. In this paper, we propose a novel graph-based approach\nfor Visual Question Answering. Our method combines a graph learner module,\nwhich learns a question specific graph representation of the input image, with\nthe recent concept of graph convolutions, aiming to learn image representations\nthat capture question specific interactions. We test our approach on the VQA v2\ndataset using a simple baseline architecture enhanced by the proposed graph\nlearner module. We obtain promising results with 66.18% accuracy and\ndemonstrate the interpretability of the proposed method. Code can be found at\ngithub.com/aimbrain/vqa-project.\n