QIS-GAN: A Lightweight Adversarial Network With Quadtree Implicit Sampling for Multispectral and Hyperspectral Image Fusion

Abstract

Multispectral and Hyperspectral Image Fusion (MHIF) involves the fusion of high spatial resolution multispectral images (HR-MSI) and low spatial resolution hyperspectral images (LR-HSI) to generate high spatial resolution hyperspectral images (HR-HSI), has gained significant attention in the field of remote sensing imaging. While CNN and Transformer models have shown effectiveness in MHIF, existing CNN or Transformer-based algorithms are overburdened with model size, making it difficult to achieve an effective trade-off between fusion accuracy and degree of lightweight. Recently, Implicit Neural Representation (INR) has been proven good interpretability and the ability to exploit coordinate information in 2D tasks. Nonetheless, INR-based fusion networks have certain limitations, such as the need for deeper super-resolution networks as shallow encoders, and insufficient representation capability on high upsampling ratios. To address these challenges, we present the Quadtree Implicit Sampling (QIS), which employs a hierarchical sampling from the perspective of the quadtree, to enhance the capacity of the overall network. Furthermore, the remarkable design of QIS allows us to adopt a lightweight structure as the shallow encoder, greatly alleviating the network burden and achieving lightweight. Inspired by generative adversarial models, we incorporate QIS as a lightweight generator into the GAN framework named QIS-GAN and leverage a discriminator to increase the fidelity of fused images. The results showcase the superior performance of QIS-GAN on the MHIF tasks with upsampling ratios of ×4, ×8, and ×16, surpassing the state-of-the-art in several datasets. The code for our approach will be available at https://github.com/chunyuzhu/QIS-GAN.