ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation

TLDR

Large‑scale semantic‑segmentation training requires dense pixel masks, which are costly to annotate, but scribbles are a widely used, user‑friendly alternative for interactive segmentation. The paper proposes to use scribbles to annotate images and train convolutional networks for semantic segmentation supervised by scribbles. The method employs a graphical model that propagates scribble information to unmarked pixels while jointly learning network parameters. The approach achieves competitive object segmentation on PASCAL VOC and excellent results on PASCALCONTEXT for unstructured classes such as water, sky, and grass. Scribble annotations for PASCAL VOC are publicly available at the provided Microsoft research URL.

Abstract

Large-scale data is of crucial importance for learning semantic segmentation models, but annotating per-pixel masks is a tedious and inefficient procedure. We note that for the topic of interactive image segmentation, scribbles are very widely used in academic research and commercial software, and are recognized as one of the most userfriendly ways of interacting. In this paper, we propose to use scribbles to annotate images, and develop an algorithm to train convolutional networks for semantic segmentation supervised by scribbles. Our algorithm is based on a graphical model that jointly propagates information from scribbles to unmarked pixels and learns network parameters. We present competitive object semantic segmentation results on the PASCAL VOC dataset by using scribbles as annotations. Scribbles are also favored for annotating stuff (e.g., water, sky, grass) that has no well-defined shape, and our method shows excellent results on the PASCALCONTEXT dataset thanks to extra inexpensive scribble annotations. Our scribble annotations on PASCAL VOC are available at http://research.microsoft.com/en-us/um/ people/jifdai/downloads/scribble_sup.