Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization

TLDR

The paper proposes a system that trains deep neural networks for object detection solely with synthetic images. The system uses domain randomization—randomizing simulator parameters such as lighting, pose, and textures—to expose the network to diverse conditions, and is evaluated on car bounding‑box detection in the KITTI dataset. The authors demonstrate that networks trained with domain‑randomized synthetic data achieve strong performance, and that fine‑tuning on real data further improves results beyond training on real data alone, showing that inexpensive synthetic data can replace costly real‑world data collection.

Abstract

We present a system for training deep neural networks for object detection using synthetic images. To handle the variability in real-world data, the system relies upon the technique of domain randomization, in which the parameters of the simulator-such as lighting, pose, object textures, etc.-are randomized in non-realistic ways to force the neural network to learn the essential features of the object of interest. We explore the importance of these parameters, showing that it is possible to produce a network with compelling performance using only non-artistically-generated synthetic data. With additional fine-tuning on real data, the network yields better performance than using real data alone. This result opens up the possibility of using inexpensive synthetic data for training neural networks while avoiding the need to collect large amounts of hand-annotated real-world data or to generate high-fidelity synthetic worlds-both of which remain bottlenecks for many applications. The approach is evaluated on bounding box detection of cars on the KITTI dataset.