Pattern Recognition · 2023 · 17 citations · 15 references
Social SciencesAudiologyNonverbal CommunicationPortraiturePsychoacousticsFacial ExpressionSpeech SynthesisAnimation DistributionAudio-driven AnimationFacial AnimationFacial Expression AnimationFacial Expression RepresentationFacial Expression SynthesisProbabilistic MappingTemporal Dependency Modeling
Audio-driven talking face generation, which aims to synthesize talking faces with realistic facial animations (including accurate lip movements, vivid facial expression details and natural head poses) corresponding to the audio, has achieved rapid progress in recent years. However, most existing work focuses on generating lip movements only without handling the closely correlated facial expressions, which degrades the realism of the generated faces greatly. This paper presents DIRFA, a novel method that can generate talking faces with diverse yet realistic facial animations from the same driving audio. To accommodate fair variation of plausible facial animations for the same audio, we design a transformer-based probabilistic mapping network that can model the variational facial animation distribution conditioned upon the input audio and autoregressively convert the audio signals into a facial animation sequence. In addition, we introduce a temporally-biased mask into the mapping network, which allows to model the temporal dependency of facial animations and produce temporally smooth facial animation sequence. With the generated facial animation sequence and a source image, photo-realistic talking faces can be synthesized with a generic generation network. Extensive experiments show that DIRFA can generate talking faces with realistic facial animations effectively.
15
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
1993 · 14.3K citations
Proceedings of IEEE International Conference on Computer Vision
1995 · 12.9K citations
Proceedings European Conference on Computer Vision 2002
Anders Heyden, Gunnar Sparr, Mads Nielsen et al. · Lund University Publications (Lund University) · 2002 · 5.2K citations