Proceedings of the 30th ACM International Conference on Multimedia · 2022 · 13 citations · 20 references
Social SciencesCommunication3D ModelingComputer VisionNonverbal CommunicationFacial ExpressionMultimodal InteractionSpeech SynthesisAudio Signal ProcessingContextEvaluation Metrics
In the task of talking head generation, it is hard to learn the mapping relationship between generated head image and input audio signal. To tackle this challenge, we propose to learn the mapping relationship between input audio signal and the parameters of three-dimensional morphable face model (3DMM) first, which is easier to learn. Then the parameters of 3DMM are used to guide the generation of high-quality talking head images. Prior works mostly encode audio features from short audio windows, which may influence the accuracy of lip movements sometimes because of the limited context. In this paper, we propose a transformer-based audio encoder to take full use of the long-term context from audio and then predict a sequence of 3DMM parameters accurately. Unlike prior works that only use the 3DMM parameters of expression, rotation and translation, we propose to include the parameters of identity. Since the location of 3D facial mesh point is decided by the expression and identity parameters, it is helpful to supply more subtle control of lip movement by considering the identity parameters. The experimental results reveal that our method ranks first in 4 of the total 11 evaluation metrics, which ranks first in the talking head generation track.
20
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh et al. · IEEE Transactions on Image Processing · 2004 · 56K citations
A 3D Face Model for Pose and Illumination Invariant Face Recognition
Pascal Paysan, Reinhard Knothe, Brian Amberg et al. · 2009 · 1.4K citations
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri et al. · 2020 · 801 citations · Full text
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
Yu Deng, Jiaolong Yang, Sicheng Xu et al. · 2019 · 742 citations