A Practical Model for Live Speech-Driven Lip-Sync

Abstract

This article introduces a simple, efficient, yet practical phoneme-based approach to generate realistic speech animation in real time based on live speech input. Specifically, the authors first decompose lower-face movements into low-dimensional principal component spaces. Then, in each of the retained principal component spaces, they select the AnimPho with the highest priority value and the minimum smoothness energy. Finally, they apply motion blending and interpolation techniques to compute final animation frames for the currently inputted phoneme. Through many experiments and comparisons, the authors demonstrate the realism of synthesized speech animation by their approach as well as its real-time efficiency on an off-the-shelf computer.