Domain randomization for transferring deep neural networks from simulation to the real world

TLDR

Bridging the reality gap between simulated robotics and real hardware can accelerate research by improving data availability, and sufficient simulator variability can make the real world appear as another variation, enabling object localization as a step toward manipulation. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. They train deep neural networks on simulated RGB images with non‑realistic random textures, using domain randomization to expose the model to diverse renderings. The resulting detector achieves 1.5 cm accuracy and robustness to distractors and partial occlusions, and can be used to grasp objects in cluttered environments, marking the first successful transfer of a deep network trained solely on simulated RGB images to real‑world robotic control.

Abstract

Bridging the `reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to 1.5 cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.