Interactformer: Interactive Transformer and CNN for Hyperspectral Image Super-Resolution

Abstract

Due to rich spectral information, hyperspectral images (HSIs) have been widely used in various fields. However, limited by imaging systems, the low spatial resolution of HSIs has become an important problem. In this article, for enhancing the spatial resolution, Interactformer is proposed to interact with global and local features extracted by Transformer and 3D convolutional neural network (CNN) branches. Within the Transformer branch, a separable self-attention module with linear complexity is designed to solve the problem that traditional self-attention mechanisms suffer from large memory costs due to quadratic complexity. In the 3D CNN branch, the spectral attention module and 3D convolution are applied jointly to better protect the spectral correlation among spectral bands and facilitate local feature extraction of HSIs. The interactive attention unit between the two parallel branches is designed to interact with local and global feature information adaptively. Compared with state-of-the-art super-resolution (SR) methods, the proposed method reconstructs better HSI in simulated SR experiments, real SR experiments, and classification experiments, which prove that Interactformer can effectively improve the spatial resolution while preserving the spectral information.