Extraction of high-resolution frames from video sequences

TLDR

Human vision can temporally integrate video frames to perceive a spatial resolution higher than that of any single frame, likely because adjacent frames contain slightly different but complementary information. The study aims to generate a single high‑resolution video frame by exploiting both spatial and temporal information in a short low‑resolution image sequence. A motion‑compensated subsampling observation model combined with Bayesian restoration using a discontinuity‑preserving prior is employed to reconstruct a high‑resolution still from the low‑resolution sequence, and the algorithm is also applied to motion‑compensated scan conversion of interlaced video. The proposed method delivers dramatic visual and quantitative improvements over bilinear, cubic B‑spline, and Bayesian single‑frame interpolations for subpixel camera pans and independently moving objects, and enhances resolution in interlaced video scan conversion.

Abstract

The human visual system appears to be capable of temporally integrating information in a video sequence in such a way that the perceived spatial resolution of a sequence appears much higher than the spatial resolution of an individual frame. While the mechanisms in the human visual system that do this are unknown, the effect is not too surprising given that temporally adjacent frames in a video sequence contain slightly different, but unique, information. This paper addresses the use of both the spatial and temporal information present in a short image sequence to create a single high-resolution video frame. A novel observation model based on motion compensated subsampling is proposed for a video sequence. Since the reconstruction problem is ill-posed, Bayesian restoration with a discontinuity-preserving prior image model is used to extract a high-resolution video still given a short low-resolution sequence. Estimates computed from a low-resolution image sequence containing a subpixel camera pan show dramatic visual and quantitative improvements over bilinear, cubic B-spline, and Bayesian single frame interpolations. Visual and quantitative improvements are also shown for an image sequence containing objects moving with independent trajectories. Finally, the video frame extraction algorithm is used for the motion-compensated scan conversion of interlaced video data, with a visual comparison to the resolution enhancement obtained from progressively scanned frames.