QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

TLDR

Grasping remains a longstanding challenge in robotic manipulation. The paper introduces QT‑Opt, a scalable self‑supervised vision‑based reinforcement learning framework that generalizes to 96 % grasp success on unseen objects. QT‑Opt trains a deep neural network Q‑function with over 1.2 M parameters on more than 580 k real‑world grasp attempts, enabling closed‑loop vision‑based control that continuously updates grasp strategy to optimize long‑horizon success. QT‑Opt achieves a 96 % grasp success rate on unseen objects and learns diverse behaviors such as regrasping, probing, repositioning, and disturbance‑.

Abstract

In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-based reinforcement learning framework that can leverage over 580k real-world grasp attempts to train a deep neural network Q-function with over 1.2M parameters to perform closed-loop, real-world grasping that generalizes to 96% grasp success on unseen objects. Aside from attaining a very high success rate, our method exhibits behaviors that are quite distinct from more standard grasping systems: using only RGB vision-based perception from an over-the-shoulder camera, our method automatically learns regrasping strategies, probes objects to find the most effective grasps, learns to reposition objects and perform other non-prehensile pre-grasp manipulations, and responds dynamically to disturbances and perturbations.