2023 · 27 citations · 30 references
ComputingRobotics EngineeringHuman-robot InteractionRoboticsSensor FusionAugmented RealityEmbodied InteractionKinematic SimulationMixed RealityMotion CaptureVirtual RealityArticulated Body SimulationFull-body Motion CaptureHuman Motion SimulationMotion TrackingMotion-based InteractionPhysics-based SimulationPose TrackingWearable Sensors
Replicating a user’s pose from only wearable sensors is important for many AR/VR applications. Most existing methods for motion tracking avoid environment interaction apart from foot-floor contact due to their complex dynamics and hard constraints. However, in daily life people regularly interact with their environment, e.g. by sitting on a couch or leaning on a desk. Using Reinforcement Learning, we show that headset and controller pose, if combined with physics simulation and environment observations can generate realistic full-body poses even in highly constrained environments. The physics simulation automatically enforces the various constraints necessary for realistic poses, instead of manually specifying them as in many kinematic approaches. These hard constraints allow us to achieve high-quality interaction motions without typical artifacts such as penetration or contact sliding. We discuss three features, the environment representation, the contact reward and scene randomization, crucial to the performance of the method. We demonstrate the generality of the approach through various examples, such as sitting on chairs, a couch and boxes, stepping over boxes, rocking a chair and turning an office chair. We believe these are some of the highest-quality results achieved for motion tracking from sparse sensor with scene interaction.
30
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke, Sam Gross, Francisco Massa et al. · arXiv (Cornell University) · 2019 · 16.2K citations · Full text
Xue Bin Peng, Pieter Abbeel, Sergey Levine et al. · ACM Transactions on Graphics · 2018 · 803 citations · Full text
Phase-functioned neural networks for character control
Daniel Holden, Taku Komura, Jun Saito · ACM Transactions on Graphics · 2017 · 531 citations · Full text
Neural state machine for character-scene interactions
Sebastian Starke, He Zhang, Taku Komura et al. · ACM Transactions on Graphics · 2019 · 279 citations · Full text
Motion reconstruction using sparse accelerometer data
Jochen Tautges, Arno Zinke, Björn Krüger et al. · ACM Transactions on Graphics · 2011 · 203 citations