Mamba Collaborative Implicit Neural Representation for Hyperspectral and Multispectral Remote Sensing Image Fusion

Abstract

Hyperspectral remote sensing images (HSIs) capture detailed spectral characteristics of features, while multispectral remote sensing images (MSIs) provide clear spatial distribution. Fusing these two types of images can enhance feature identification and classification accuracy. Current deep learning algorithms achieve high fusion quality but struggle with balancing global effective perception and lightweight computation. Moreover, these algorithms typically discretely handle data mapping, which contrasts with the continuous nature of the world. Recently, the Mamba has shown significant potential for complex long-range modeling, addressing the computational complexity of global perception. Concurrently, implicit neural representation (INR) offers high-quality solutions for continuous domain modeling. To this end, this study introduces a novel network architecture that combines Mamba and INR, termed the Mamba cooperative INR fusion network (MCIFNet). MCIFNet effectively captures global image information and generates fused images in a continuous domain through point-to-point processing. The network comprises two main units: potential space projection and semantic extraction and fusion. The potential space projection unit performs shallow encoding of hyperspectral and MSIs, mapping them to a latent feature space. The semantic extraction and fusion unit (SEFU) uses scale adaptive residual state spatial and implicit spatial-spectral fusion (ISSF) modules to extract deep features from the bimodal images, generating fused images point-by-point. A series of fusion experiments with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula>, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8\times $ </tex-math></inline-formula>, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$16\times $ </tex-math></inline-formula> scale factors demonstrate that MCIFNet surpasses popular algorithms in both spatial detail and spectral information reconstruction, while also providing more lightweight performance. The code for MCIFNet will be shared on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/chunyuzhu/MCIFNet</uri>.