One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

TLDR

The authors propose a neural talking‑head video synthesis model for video conferencing. The model synthesizes a talking‑head video from a source image and a driving video, using a novel unsupervised keypoint representation that separates identity and motion. Experiments show the model outperforms state‑of‑the‑art methods, achieves H.264‑level visual quality at one‑tenth the bandwidth, and supports head rotation for realistic video‑conferencing.

Abstract

We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person’s appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.