A new frame interpolation scheme for talking head sequences

Abstract

A video codec typically-skips frames to satisfy the bit rate constraint. This results in a jerky motion and a loss of lip synchronization in sequences of talking persons. We propose a frame interpolation technique to solve these problems. The main components of this technique include: foreground/background segmentation, mesh-based frame interpolation, and image analysis/synthesis for mouth movements. This technique generates smooth head motion and renders lip motion that is synchronized with the voice of the person.