Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

TLDR

Imitation learning is a powerful paradigm for robot skill acquisition, but obtaining pixel‑based demonstrations suitable for learning a policy that maps raw pixels to actions can be challenging. The paper demonstrates that consumer‑grade VR headsets and hand‑tracking hardware can be used to naturally teleoperate robots for complex tasks. The method trains deep neural networks via imitation learning to map raw pixel observations to robot actions, leveraging VR‑based teleoperation demonstrations. Experiments demonstrate that the approach effectively learns visuomotor skills.

Abstract

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.