2023 · 45 citations · 38 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingConversational User InterfacesFacial ExpressionMultimodal InteractionSelf-supervised LearningSpeech RecognitionSpeech SynthesisAudio Self-supervised LearningAudio-visual Speech RecognitionCross-modal Self-supervisionLabelsMultimodal Self-supervised LearningSelf-supervised Representation LearningShapeSpeech-to-animation Mapping
Speech-driven 3D face animation technique, extending its applications to various multimedia fields.Previous research has generated promising realistic lip movements and facial expressions from audio signals. However, traditional regression models solely driven by data face several essential problems, such as difficulties in accessing precise labels and domain gaps between different modalities, leading to unsatisfactory results lacking precision and coherence.To enhance the visual accuracy of generated lip movement while reducing the dependence on labeled data, we propose a novel framework SelfTalk, by involving self-supervision in a cross-modals network system to learn 3D talking faces. The framework constructs a network system consisting of three modules: facial animator, speech recognizer, and lip-reading interpreter. The core of SelfTalk is a commutative training diagram that facilitates compatible features exchange among audio, text, and lip shape, enabling our models to learn the intricate connection between these factors. The proposed framework leverages the knowledge learned from the lip-reading interpreter to generate more plausible lip shapes. Extensive experiments and user studies demonstrate that our proposed approach achieves state-of-the-art performance both qualitatively and quantitatively. We recommend watching the supplementary video.
38
Laurens van der Maaten, Geoffrey E. Hinton · Journal of Machine Learning Research · 2008 · 35.7K citations
Connectionist temporal classification
Alex Graves, Santiago Fernández, Faustino Gomez et al. · 2006 · 5.4K citations
Learning a model of facial shape and expression from 4D scans
Tianye Li, Timo Bolkart, Michael J. Black et al. · ACM Transactions on Graphics · 2017 · 1.1K citations
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri et al. · 2020 · 801 citations · Full text
Lip Reading Sentences in the Wild
2017 · 595 citations · Full text