MTC-CSNet: Marrying Transformer and Convolution for Image Compressed Sensing

Abstract

Image compressed sensing (ICS) has been extensively applied in various imaging domains due to its capability to sample and reconstruct images at subNyquist sampling rates. The current predominant approaches in ICS, specifically pure convolutional networks (ConvNets)-based ICS methods, have demonstrated their effectiveness in capturing local features for image recovery. Simultaneously, the Transformer architecture has gained significant attention due to its capability to model global correlations among image features. Motivated by these insights, we propose a novel hybrid network for ICS, named MTC-CSNet, which effectively combines the strengths of both ConvNets and Transformer architectures in capturing local and global image features to achieve high-quality image recovery. Particularly, MTC-CSNet is a dual-path framework that consists of a ConvNets-based recovery branch and a Transformer-based recovery branch. Along the ConvNets-based recovery branch, we design a lightweight scheme to capture the local features in natural images. Meanwhile, we implement a Transformer-based recovery branch to iteratively model the global dependencies among image patches. Ultimately, the ConvNets-based and Transformer-based recovery branches collaborate through a bridging unit, facilitating the adaptive transmission and fusion of informative features for image reconstruction. Extensive experimental results demonstrate that our proposed MTC-CSNet surpasses the state-of-the-art methods on various public datasets. The code and models are publicly available at MTC-CSNet.