2024 · 13 citations · 33 references
ComputingLinguisticsComputer VisionPhonologyPsychoacousticsDeep LearningMultimodal CorporaMultimodal InteractionSpeech RecognitionAudio-visual CorporaAudio-visual Speech RecognitionBackground Noise RobustnessMultilingual Speech RecognitionNoise RobustnessRobust Speech RecognitionRobustness To Noise
Humans are adept at leveraging visual cues from lip movements for recognizing speech in adverse listening conditions. Audio-Visual Speech Recognition (AVSR) models follow similar approach to achieve robust speech recognition in noisy conditions. In this work, we present a multilingual AVSR model incorporating several enhancements to improve performance and audio noise robustness. Notably, we adapt the recently proposed Fast Conformer model to process both audio and visual modalities using a novel hybrid CTC/RNN-T architecture. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets (VoxCeleb2 and AVSpeech). Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%. On the recently introduced MuAViC benchmark, our model yields an absolute average-WER reduction of 11.9% in comparison to the original baseline. Finally, we demonstrate the ability of the proposed model to perform audio-only, visual-only, and audio-visual speech recognition at test time.
33
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter · arXiv (Cornell University) · 2017 · 9.1K citations · Full text
Connectionist temporal classification
Alex Graves, Santiago Fernández, Faustino Gomez et al. · 2006 · 5.4K citations
Conformer: Convolution-augmented Transformer for Speech Recognition
Anmol Gulati, James Qin, Chung‐Cheng Chiu et al. · 2020 · 2.7K citations