IEEE Signal Processing Letters · 2022 · 23 citations · 9 references
ComputingAutonomous SystemsAcousticsMixed MediaPsychoacousticsMultimodal InteractionPlayback SystemsAudio InterfacesAudio-visual AlignmentAudio-visual RepresentationAudio-visual SystemsAudiovisual InterfacesDigital Audio Systems
Conventional audio-visual models have independent audio and video branches. In this work, we <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unify</i> the audio and visual branches by designing a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</u> nified <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</u> udio- <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</u> isual <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</u> odel (UAVM). The UAVM achieves a new state-of-the-art audio-visual event classification accuracy of 65.8% on VGGSound. More interestingly, we also find a few intriguing properties of UAVM that the modality-independent counterparts do not have.
9
Audio Set: An ontology and human-labeled dataset for audio events
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman et al. · 2017 · 3K citations
AST: Audio Spectrogram Transformer
Yuan Gong, Yu-An Chung, James Glass · 2021 · 989 citations
Relja Arandjelović, Andrew Zisserman · 2017 · 765 citations
What Makes Training Multi-Modal Classification Networks Hard?
Wei‐Yao Wang, Du Tran, Matt Feiszli · 2020 · 485 citations