Proceedings of the AAAI Conference on Artificial Intelligence · 2020 · 180 citations · 32 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingDeep LearningGraph-based LearningMulti-task LearningNatural Language UnderstandingQuestion AnsweringVisual Question AnsweringAttentionBenchmark DatasetsCross-modal Representation LearningHeterogeneous Graph LearningLate Fusion ArchitecturesMultimodal FusionMultimodal Representation LearningVideo Question Answering
The dominant video question answering methods are based on fine-grained representation or model-specific attention mechanism. They usually process video and question separately, then feed the representations of different modalities into following late fusion networks. Although these methods use information of one modality to boost the other, they neglect to integrate correlations of both inter- and intra-modality in an uniform module. We propose a deep heterogeneous graph alignment network over the video shots and question words. Furthermore, we explore the network architecture from four steps: representation, fusion, alignment, and reasoning. Within our network, the inter- and intra-modality information can be aligned and interacted simultaneously over the heterogeneous graph and used for cross-modal reasoning. We evaluate our method on three benchmark datasets and conduct extensive ablation study to the effectiveness of the network architecture. Experiments show the network to be superior in quality.
32
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Learning Spatiotemporal Features with 3D Convolutional Networks
Du Tran, Lubomir Bourdev, Rob Fergus et al. · 2015 · 9.7K citations