IMoS: Intent‐Driven Full‐Body Motion Synthesis for Human‐Object Interactions

TLDR

Virtual character motion synthesis that incorporates object interactions remains largely unexplored, as most prior work focuses only on hand or finger movements and ignores full‑body dynamics. This paper presents the first framework to synthesize full‑body motion of virtual humans performing specified actions with 3D objects placed within their reach. The framework employs intent‑driven full‑body motion generators built from paired conditional variational auto‑regressors that learn body‑part motion autoregressively, taking textual instructions about objects and intentions as input while simultaneously optimizing the objects’ 6‑DoF poses to fit the characters’ hands. The method outperforms existing motion synthesis approaches, establishing a new state‑of‑the‑art for intent‑driven full‑body motion synthesis.

Abstract

Abstract Can we make virtual characters in a scene interact with their surrounding objects through simple instructions? Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions? Inspired by these questions, we present the first framework to synthesize the full‐body motion of virtual human characters performing specified actions with 3D objects placed within their reach. Our system takes textual instructions specifying the objects and the associated ‘intentions’ of the virtual characters as input and outputs diverse sequences of full‐body motions. This contrasts existing works, where full‐body action synthesis methods generally do not consider object interactions, and human‐object interaction methods focus mainly on synthesizing hand or finger movements for grasping objects. We accomplish our objective by designing an intent‐driven full‐body motion generator, which uses a pair of decoupled conditional variational auto‐regressors to learn the motion of the body parts in an autoregressive manner. We also optimize the 6‐DoF pose of the objects such that they plausibly fit within the hands of the synthesized characters. We compare our proposed method with the existing methods of motion synthesis and establish a new and stronger state‐of‐the‐art for the task of intent‐driven motion synthesis.