3d video generation and service based on a TOF depth sensor in MPEG-4 multimedia framework

Abstract

In this paper, we present a new method to generate and serve 3D video represented by video-plus-depth using a time-of-flight (TOF) depth sensor. In practice, depth images captured by the depth sensor have critical problems, such as optical noise, unmatched boundaries with their corresponding color images, and depth flickering artifacts in the temporal domain. In this work, we enhance the noisy depth images by performing a series of processing steps including joint bilateral filtering with inner-edge selection, outer-boundary refinement by a robust image matting method, and temporal consistency based on motion estimation. Thereafter, the generated high-quality video-plus-depth is combined with computer graphics models in the MPEG-4 multimedia framework. Finally, the immersive video content is streamed to consumers to enjoy 3D view. Experimental results show that our method can minimize the inherent problems of depth images significantly and serve 3D video successfully in the MPEG-4 multimedia framework.