Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

TLDR

Deep Convolutional Neural Networks have achieved state‑of‑the‑art performance in high‑level vision tasks such as classification and detection, largely because their strong invariance properties benefit these tasks. The study aims to combine DCNNs with probabilistic graphical models to perform pixel‑level semantic image segmentation. The authors fuse final‑layer DCNN responses with a fully connected CRF and employ network re‑purposing plus a wavelet‑inspired “hole” algorithm to compute dense responses at 8 fps on a GPU. The DeepLab system demonstrates that DCNN final‑layer responses lack sufficient localization, yet by integrating a CRF it achieves superior boundary accuracy and sets a new PASCAL VOC‑2012 state‑of‑the‑art with 71.6 % IOU.

Abstract

Deep Convolutional Neural Networks (DCNNs) have recently shown state of the art performance in high level vision tasks, such as image classification and object detection. This work brings together methods from DCNNs and probabilistic graphical models for addressing the task of pixel-level classification (also called "semantic image segmentation"). We show that responses at the final layer of DCNNs are not sufficiently localized for accurate object segmentation. This is due to the very invariance properties that make DCNNs good for high level tasks. We overcome this poor localization property of deep networks by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF). Qualitatively, our "DeepLab" system is able to localize segment boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set. We show how these results can be obtained efficiently: Careful network re-purposing and a novel application of the 'hole' algorithm from the wavelet community allow dense computation of neural net responses at 8 frames per second on a modern GPU.