2021 · 44 citations · 27 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingMultimodal InteractionSelf-supervised LearningSpeech RecognitionSupervised LearningAudio Self-supervised LearningAudio-visual Speech RecognitionCross-modal Self-supervisionFine-tuningMultimodal Self-supervised LearningSelf-supervised PretrainingSelf-supervised Representation LearningSelf-supervised Speech Representation Learning
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning.Recent works have focused on each of these modalities separately, while others have attempted to model both simultaneously in a cross-modal fashion.However, comparatively little attention has been given to leveraging one modality as a training objective to learn from the other.In this work, we propose Learning visual speech Representations from Audio via self-supervision (LiRA).Specifically, we train a ResNet+Conformer model to predict acoustic features from unlabelled visual speech.We find that this pre-trained model can be leveraged towards word-level and sentence-level lip-reading through feature extraction and fine-tuning experiments.We show that our approach significantly outperforms other self-supervised methods on the Lip Reading in the Wild (LRW) dataset and achieves state-of-the-art performance on Lip Reading Sentences 2 (LRS2) using only a fraction of the total labelled data.
27
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter · arXiv (Cornell University) · 2017 · 9.1K citations · Full text
Librispeech: An ASR corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey et al. · 2015 · 6K citations
Computing, Computer Science, English +17
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
Zihang Dai, Zhilin Yang, Yiming Yang et al. · 2019 · 3.1K citations · Full text