Video Object Segmentation: A Compressed Domain Approach

TLDR

Object segmentation in MPEG relies solely on sparse motion vectors, and the method is applicable to object‑based coding in MPEG‑4. The paper proposes a method to automatically estimate the number of objects and extract independently moving video objects from MPEG compressed video using motion vectors. Motion vectors are accumulated over several frames, spatially interpolated to dense vectors, then used in an EM algorithm with block‑based affine clustering to determine motion models and segment objects, followed by edge refinement for precise boundaries. Illustrative examples show the effectiveness of the proposed approach.

Abstract

This paper addresses the problem of extracting video objects from MPEG compressed video. The only cues used for object segmentation are the motion vectors which are sparse in MPEG. A method for automatically estimating the number of objects and extracting independently moving video objects using motion vectors is presented here. First, the motion vectors are accumulated over a few frames to enhance the motion information, which are further spatially interpolated to get dense motion vectors. The final segmentation, using the dense motion vectors, is obtained by applying the expectation maximization (EM) algorithm. A block-based affine clustering method is proposed for determining the number of appropriate motion models to be used for the EM step and the segmented objects are temporally tracked to obtain the video objects. Finally, a strategy for edge refinement is proposed to extract the precise object boundaries. Illustrative examples are provided to demonstrate the efficacy of the approach. A prominent application of the proposed method is that of object-based coding, which is part of the MPEG-4 standard.