Computational Visual Media · 2023 · 15 citations · 44 references
Social SciencesVision ScienceEye MovementsMusic AnalysisNonverbal CommunicationFacial ExpressionMultimodal InteractionSpeech SynthesisAttentionBlinking DynamicsDynamic Facial ExpressionsFacial Expression AnalysisFacial Expression DynamicsFacial Expression SynthesisHead Pose EstimationMovement Generation
It remains an interesting and challenging problem to synthesize a vivid and realistic singing face driven by music. In this paper, we present a method for this task with natural motions for the lips, facial expression, head pose, and eyes. Due to the coupling of mixed information for the human voice and backing music in common music audio signals, we design a decouple-and-fuse strategy to tackle the challenge. We first decompose the input music audio into a human voice stream and a backing music stream. Due to the implicit and complicated correlation between the two-stream input signals and the dynamics of the facial expressions, head motions, and eye states, we model their relationship with an attention scheme, where the effects of the two streams are fused seamlessly. Furthermore, to improve the expressivenes of the generated results, we decompose head movement generation in terms of speed and direction, and decompose eye state generation into short-term blinking and long-term eye closing, modeling them separately. We have also built a novel dataset, SingingFace, to support training and evaluation of models for this task, including future work on this topic. Extensive experiments and a user study show that our proposed method is capable of synthesizing vivid singing faces, qualitatively and quantitatively better than the prior state-of-the-art.
44
Squeeze-and-Excitation Networks
Least Squares Generative Adversarial Networks
Xudong Mao, Qing Li, Haoran Xie et al. · 2017 · 5.2K citations
VoxCeleb2: Deep Speaker Recognition
Joon Son Chung, Arsha Nagrani, Andrew Zisserman · 2018 · 2.3K citations · Full text
Proceedings of The 32nd International Conference on Machine Learning
Mathieu Germain, Karol Gregor, Iain Murray et al. · 2015 · 2K citations