Hyperspectral Image Super-Resolution With ConvLSTM Skip-Connections

Abstract

Hyperspectral image super-resolution has been extensively studied, and significant development has been made based on deep convolutional neural networks (CNNs). Particularly, residual networks that fuse features from multiple layers have achieved high-accuracy hyperspectral image super-resolution. However, most residual networks tend to straightforwardly add features from one layer to another through skip-connections that may cause confusion about feature fusion. To tackle this issue, we develop a ConvLSTM skip-connection strategy that characterizes features from consecutive layers by ConvLSTMs and renders feature fusion in a more principal manner. Accordingly, we develop a super-resolution framework that consists of three modules. The first module, i.e., spatial feature reconstruction, employs the ConvLSTM skip-connections to comprehensively fuse spatial features from different layers. The second module, i.e., edge refinement, involves the ConvLSTM skip-connections to enhance the edge information from intermediate results. The third module, i.e., spectral information reconstruction, refines spectral features by capturing interactions between different spectral bands through the ConvLSTM skip-connections. The three complementary modules cooperate such that both spatial resolution and spectral fidelity are well maintained. Extensive experimental results on the Chikusei, Houston, and QUST-1 datasets demonstrate that our framework outperforms state-of-the-art methods in terms of quantitative evaluation and visual quality across a variety of scenarios. We release our source code at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://gitee.com/xu_yinghao/CLSCNet</uri> for public evaluations.