SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips

TLDR

Unlike offline systems that require hours or days for capture, labeling, and batch learning, our approach operates fully online. We present an interactive, online approach to 3D scene understanding that also enables the capture of large labeled 3D datasets for training visual recognition systems. SemanticPaint lets users scan and interactively segment scenes by touching objects, while continuously learning from those segmentations to label unseen parts of the environment. The system offers continuous live feedback, enabling immediate error correction, and produces personalized models that can be applied to augmented reality, interior design, and navigation.

Abstract

We present a new interactive and online approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment whilst interactively segmenting the scene simply by reaching out and touching any desired object or surface. Our system continuously learns from these segmentations, and labels new unseen parts of the environment. Unlike offline systems where capture, labeling, and batch learning often take hours or even days to perform, our approach is fully online. This provides users with continuous live feedback of the recognition during capture, allowing to immediately correct errors in the segmentation and/or learning—a feature that has so far been unavailable to batch and offline methods. This leads to models that are tailored or personalized specifically to the user's environments and object classes of interest, opening up the potential for new applications in augmented reality, interior design, and human/robot navigation. It also provides the ability to capture substantial labeled 3D datasets for training large-scale visual recognition systems.