Feature Space Optimization for Semantic Video Segmentation

TLDR

Temporal regularization in video is challenging because both camera and scene motion make Euclidean distance in the space‑time volume a poor proxy for correspondence. The study proposes optimizing pixel mapping to a Euclidean feature space to enable long‑range spatio‑temporal regularization in semantic video segmentation. The method optimizes pixel mapping to a Euclidean feature space and then applies a dense CRF for structured prediction on the optimized features. Experimental results demonstrate that the presented approach increases the accuracy and temporal consistency of semantic video segmentation.

Abstract

We present an approach to long-range spatio-temporal regularization in semantic video segmentation. Temporal regularization in video is challenging because both the camera and the scene may be in motion. Thus Euclidean distance in the space-time volume is not a good proxy for correspondence. We optimize the mapping of pixels to a Euclidean feature space so as to minimize distances between corresponding points. Structured prediction is performed by a dense CRF that operates on the optimized features. Experimental results demonstrate that the presented approach increases the accuracy and temporal consistency of semantic video segmentation.