2024 · 119 citations · 32 references
ComputingPerforming ArtsComputer VisionGenerative ArtNonverbal CommunicationFacial ExpressionGenerative ModelingImage SynthesisSpeech SynthesisAutoregressive ModelsAutoregressive ProcessesDiffusion ModelsVideo Generation
Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments in diffusion-based generative models allow for more realistic and stable data synthesis and their performance on image and video generation has surpassed that of other generative models. In this work, we present an autoregressive diffusion model that requires only one identity image and audio sequence to generate a video of a realistic talking head. Our solution is capable of hallucinating head movements, facial expressions, such as blinks, and preserving a given background. We evaluate our model on two different datasets, achieving state-of-the-art results in expressiveness and smoothness on both of them. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
32
Generative adversarial networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza et al. · Communications of the ACM · 2020 · 13.6K citations · Full text
Denoising Diffusion Probabilistic Models
arXiv (Cornell University) · 2020 · 5.6K citations · Full text
Hierarchical Text-Conditional Image Generation with CLIP Latents
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol et al. · arXiv (Cornell University) · 2022 · 2.3K citations · Full text
Diffusion Models Beat GANs on Image Synthesis
Prafulla Dhariwal, Alex Nichol · arXiv (Cornell University) · 2021 · 2.2K citations · Full text