Gesture modeling and animation based on a probabilistic re-creation of speaker style

TLDR

Animated characters that move and gesticulate appropriately with spoken text are useful across many applications, yet generating such movement—especially in a unique individual style—is notoriously difficult. The authors present a system that, focusing on arm gestures, can produce full‑body gesture animation for given input text in the style of a particular performer. The system builds a statistical model of a performer’s gesturing style from annotated video, then uses that model together with text tags to generate a continuous gesture script that an animation engine converts into kinematic or physically simulated motion. The system is capable of generating gesture animations for novel text that are consistent with a given performer’s style, as validated in an empirical user study.

Abstract

Animated characters that move and gesticulate appropriately with spoken text are useful in a wide range of applications. Unfortunately, this class of movement is very difficult to generate, even more so when a unique, individual movement style is required. We present a system that, with a focus on arm gestures, is capable of producing full-body gesture animation for given input text in the style of a particular performer. Our process starts with video of a person whose gesturing style we wish to animate. A tool-assisted annotation process is performed on the video, from which a statistical model of the person's particular gesturing style is built. Using this model and input text tagged with theme, rheme and focus, our generation algorithm creates a gesture script. As opposed to isolated singleton gestures, our gesture script specifies a stream of continuous gestures coordinated with speech. This script is passed to an animation system, which enhances the gesture description with additional detail. It then generates either kinematic or physically simulated motion based on this description. The system is capable of generating gesture animations for novel text that are consistent with a given performer's style, as was successfully validated in an empirical user study.