Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data

TLDR

Segmenting unknown objects in depth images can improve robotic grasping and tracking, and Mask R‑CNN has proven effective for category‑specific segmentation in RGB images, while synthetic depth data has shown promising transfer to real‑world depth sensors. The study aims to train Mask R‑CNN on synthetic depth images to avoid the time‑consuming creation of hand‑labeled datasets. The authors automatically generate 50,000 synthetic depth images with 320,000 masks from simulated CAD model heaps, train a domain‑randomized Mask R‑CNN variant on this data, evaluate it on real high‑resolution depth images of cluttered bins, and integrate the model into a grasping pipeline. SD Mask R‑CNN surpasses point‑cloud clustering baselines by 15 % AP and 20 % AR on COCO, matches the performance of a Mask R‑CNN trained on a massive hand‑labeled RGB dataset fine‑tuned on real images, and the code and synthetic dataset are publicly released.

Abstract

The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive hand-labeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can transfer successfully to the real world. We present a method for automated dataset generation and rapidly generate a synthetic training dataset of 50,000 depth images and 320,000 object masks using simulated heaps of 3D CAD models. We train a variant of Mask R-CNN with domain randomization on the generated dataset to perform category-agnostic instance segmentation without any hand-labeled data and we evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution depth images of challenging, densely-cluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall on COCO benchmarks, and achieves performance levels similar to a Mask R-CNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE.