Bilateral Space Video Segmentation

TLDR

The study proposes a novel video segmentation method operating in bilateral space. The method formulates an energy on a spatiotemporal bilateral grid and solves it with graph cuts, enabling efficient long‑range spatio‑temporal connections with few variables. The approach outperforms recent methods, achieving state‑of‑the‑art results on multiple benchmarks while running faster and scaling linearly with image size for interactive high‑resolution video.

Abstract

In this work, we propose a novel approach to video segmentation that operates in bilateral space. We design a new energy on the vertices of a regularly sampled spatiotemporal bilateral grid, which can be solved efficiently using a standard graph cut label assignment. Using a bilateral formulation, the energy that we minimize implicitly approximates long-range, spatio-temporal connections between pixels while still containing only a small number of variables and only local graph edges. We compare to a number of recent methods, and show that our approach achieves state-of-the-art results on multiple benchmarks in a fraction of the runtime. Furthermore, our method scales linearly with image size, allowing for interactive feedback on real-world high resolution video.