ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene\n Understanding Using Mobile RGB-D Data

Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter P. Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz,

arXiv (Cornell University) · 2021 · 33 citations · 0 references

DOIFull text

Open access

Concepts

Abstract

Scene understanding is an active research area. Commercial depth sensors,\nsuch as Kinect, have enabled the release of several RGB-D datasets over the\npast few years which spawned novel methods in 3D scene understanding. More\nrecently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high\nquality RGB-D data is accessible to millions of people on a device they\ncommonly use. This opens a whole new era in scene understanding for the\nComputer Vision community as well as app developers. The fundamental research\nin scene understanding together with the advances in machine learning can now\nimpact people's everyday experiences. However, transforming these scene\nunderstanding methods to real-world experiences requires additional innovation\nand development. In this paper we introduce ARKitScenes. It is not only the\nfirst RGB-D dataset that is captured with a now widely available depth sensor,\nbut to our best knowledge, it also is the largest indoor scene understanding\ndata released. In addition to the raw and processed data from the mobile\ndevice, ARKitScenes includes high resolution depth maps captured using a\nstationary laser scanner, as well as manually labeled 3D oriented bounding\nboxes for a large taxonomy of furniture. We further analyze the usefulness of\nthe data for two downstream tasks: 3D object detection and color-guided depth\nupsampling. We demonstrate that our dataset can help push the boundaries of\nexisting state-of-the-art methods and it introduces new challenges that better\nrepresent real-world scenarios.\n