Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set

TLDR

The authors introduce the Yale‑CMU‑Berkeley (YCB) object and model set and an accompanying framework to enable standardized benchmarking in robotic manipulation research. The YCB set comprises diverse daily‑life objects with varied shapes, sizes, textures, weights, and rigidities, along with high‑resolution RGB‑D scans, physical properties, and geometric models, and is supported by a literature survey and protocol templates for quantitative evaluation. The availability of the YCB set and its protocols is expected to facilitate easier comparison of manipulation approaches and drive the evolution of standardized benchmarks and metrics in the field.

Abstract

In this article, we present the Yale-Carnegie Mellon University (CMU)-Berkeley (YCB) object and model set, intended to be used to facilitate benchmarking in robotic manipulation research. The objects in the set are designed to cover a wide range of aspects of the manipulation problem. The set includes objects of daily life with different shapes, sizes, textures, weights, and rigidities as well as some widely used manipulation tests. The associated database provides high-resolution red, green, blue, plus depth (RGB-D) scans, physical properties, and geometric models of the objects for easy incorporation into manipulation and planning software platforms. In addition to describing the objects and models in the set along with how they were chosen and derived, we provide a framework and a number of example task protocols, laying out how the set can be used to quantitatively evaluate a range of manipulation approaches, including planning, learning, mechanical design, control, and many others. A comprehensive literature survey on the existing benchmarks and object data sets is also presented, and their scope and limitations are discussed. The YCB set will be freely distributed to research groups worldwide at a series of tutorials at robotics conferences. Subsequent sets will be, otherwise, available to purchase at a reasonable cost. It is our hope that the ready availability of this set along with the ground laid in terms of protocol templates will enable the community of manipulation researchers to more easily compare approaches as well as continually evolve standardized benchmarking tests and metrics as the field matures.