Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

TLDR

Deep reinforcement learning can learn complex robotic skills from raw sensory inputs, yet it has not yet matched the broad generalization and applicability of supervised deep learning methods. The authors introduce a practical deep RL method for real‑world robotic manipulation that generalizes to never‑before‑seen tasks and objects. Their approach trains a self‑supervised, model‑based predictive model on autonomous robot data to forecast future camera images, and at test time permits goal specification through designated pixels, goal images, or image classifiers. The resulting visual MPC system generalizes to unseen rigid and deformable objects and solves a variety of user‑defined manipulation tasks with a single model.

Abstract

Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly predict the future from raw sensory readings, such as camera images. At test time, we explore three distinct goal specification methods: designated pixels, where a user specifies desired object manipulation tasks by selecting particular pixels in an image and corresponding goal positions, goal images, where the desired goal state is specified with an image, and image classifiers, which define spaces of goal states. Our deep predictive models are trained using data collected autonomously and continuously by a robot interacting with hundreds of objects, without human supervision. We demonstrate that visual MPC can generalize to never-before-seen objects---both rigid and deformable---and solve a range of user-defined object manipulation tasks using the same model.