2022 IEEE International Conference on Image Processing (ICIP) · 2022 · 11 citations · 11 references
ComputingComputer ScienceComputer VisionMachine LearningVisual AttentionDeep LearningFeature-based AttentionNatural Language UnderstandingQuestion AnsweringScene UnderstandingVisual Question AnsweringAttentionAttention MechanismsDisaster Damage AssessmentExplainable Visual Question AnsweringPost-disaster Damage AssessmentTrainingVisual Explanations
In this paper, we present a Grad-Cam aware supervised attention framework for visual question answering (VQA) tasks for post-disaster damage assessment purposes. Visual-attention in visual question-answering tasks aims to focus on relevant image regions according to questions to predict answers. However, the conventional attention mechanisms in VQA work in an unsupervised manner, learning to give importance to visual contents by minimizing only task-specific loss. This approach fails to provide appropriate visual attention where the visual contents are very complex. The content and nature of UAV images in FlooNet-VQA dataset are very complex as they depict the hazardous scenario after Hurricane Harvey from a high altitude. To tackle this, we propose a supervised attention mechanism that uses explainable features from Grad-Cam to supervise visual attention in the VQA pipeline. The mechanism we propose operates in two stages. In the first stage of learning, we derived the visual explanations through Grad-Cam by training a baseline attention-based VQA model. In the second stage, we supervise our visual content for each question by incorporating the Grad-Cam explanations from the previous phase of the training process. We have improved the model performance over the state-of-the-art VQA models by a considerable margin on FloodNet dataset.
11
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das et al. · 2017 · 21.6K citations
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Peter Anderson, Xiaodong He, Chris Buehler et al. · 2018 · 5.1K citations
VQA: Visual Question Answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu et al. · 2015 · 4.3K citations
Stacked Attention Networks for Image Question Answering
Zichao Yang, Xiaodong He, Jianfeng Gao et al. · 2016 · 2.1K citations
Hierarchical Question-Image Co-Attention for Visual Question Answering
Jiasen Lu, Jianwei Yang, Dhruv Batra et al. · arXiv (Cornell University) · 2016 · 1.2K citations · Full text