Computer Vision and Image Understanding

Wael Saideni, Fabien Courrèges, David Helbert, Jean‐Pierre Cances

SSRN Electronic Journal · 2022 · 768 citations · 32 references

DOIFull text

Open access

Concepts

TL;DR

The study introduces an end‑to‑end Video Compressive Sensing framework that captures long‑range spatio‑temporal relations using a video Transformer. The authors propose ViT‑SCI, a Video Transformer employing Spatio‑Temporal Convolutional Multi‑Head Attention for snapshot compressive imaging, and conduct an ablation study of the Transformer network. ViT‑SCI outperforms prior state‑of‑the‑art methods, delivering high‑quality 64×64 video reconstruction at an unprecedented 1 frame per ms.

Abstract

In this work, we study a complete framework of Video Compressive Sensing (VCS), from capturing a sequence of video frames in one single compressed measurement to reconstructing the original frames. To our best knowledge, we present the first end-to-end sampling and recovery network built upon video Transformers, widely explored in vision related tasks, to capture long-range spatio-temporal relations. Our proposed Video Transformer for Snapshot Compressive Imaging recovery (ViT-SCI) is based on Spatio-temporal Convolutional Multi-Head Attention (ST-ConvMHA) which is an extended version of the fully-connected attention adapted for vision problems. Our comprehensive qualitative and quantitative experiments on several datasets demonstrate that ViT-SCI outperforms previous state-of-the-art methods with much faster reconstruction capacities, which pave the way for applying our algorithm in real-time applications. Indeed, ViT-SCI achieves high quality reconstruction on 64 × 64 video frames at the unprecedented rate of 1 frame per ms. In addition, an important ablation study on the Transformer network is provided to inspire future research works aiming to test the abilities of Transformers in vision tasks.

References

32