Proceedings of the AAAI Conference on Artificial Intelligence · 2020 · 102 citations · 17 references
MedicineComputer ScienceComputer VisionVisual AttentionVisual PerceptionAttention NetworksTemporal AttentionAttention-based FusionAudio-visual PerceptionCross-modal Attention In PerceptionCross-modal LocalizationInconsistency HandlingMulti-modal PerceptionMultimodal FusionPerception Systems
In human multi-modality perception systems, the benefits of integrating auditory and visual information are extensive as they provide plenty supplementary cues for understanding the events. Despite some recent methods proposed for such application, they cannot deal with practical conditions with temporal inconsistency. Inspired by human system which puts different focuses at specific locations, time segments and media while performing multi-modality perception, we provide an attention-based method to simulate such process. Similar to human mechanism, our network can adaptively select “where” to attend, “when” to attend and “which” to attend for audio-visual event localization. In this way, even with large temporal inconsistent between vision and audio, our network is able to adaptively trade information between different modalities and successfully achieve event localization. Our method achieves state-of-the-art performance on AVE (Audio-Visual Event) dataset collected in the real life. In addition, we also systemically investigate audio-visual event localization tasks. The visualization results also help us better understand how our model works.
17
Learning Deep Features for Discriminative Localization
Bolei Zhou, Aditya Khosla, Àgata Lapedriza et al. · 2016 · 10.8K citations · Full text
Audio Set: An ontology and human-labeled dataset for audio events
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman et al. · 2017 · 3K citations
Temporal Convolutional Networks for Action Segmentation and Detection
Colin Lea, M. D. Flynn, René Vidal et al. · 2017 · 2K citations
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Jiasen Lu, Caiming Xiong, Devi Parikh et al. · 2017 · 1.7K citations
A Structured Self-attentive Sentence Embedding
Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos et al. · arXiv (Cornell University) · 2017 · 1.5K citations · Full text