Self-Adaptive Global Feature Fusion Network With Spectral Prompt for Hyperspectral Image Classification

Abstract

Nowadays, foundation models have demonstrated exceptional performance across numerous downstream tasks. However, the effective application of these models to hyperspectral image classification (HSIC) is challenged by the unique characteristics of hyperspectral data, including high dimensionality, high variability, and high spatial structure complexity. Therefore, methods need to be developed, which leverage the advantages of foundation models while addressing these challenges. First, a novel HSIC algorithm based on a frozen-parameter segment anything model (SAM) encoder, called SAGFFNet, is proposed. This framework represents the first attempt to use a frozen SAM encoder for global feature extraction and to use spectral dimension data as prompts, enabling precise global spatial-spectral feature extraction with the aid of spectral information. Second, by introducing the self-adaptive padding mechanism and the global feature extraction subnetwork (GFEsNet), the model is enabled to extract distinctive and discriminative features for each category from hyperspectral data through varying padding sizes, thereby enhancing the feature extraction and generalization capabilities of the foundation model. Subsequently, the spectral feature prompt subnetwork (SFPsNet) is designed to extract spectral feature information from samples of different classes as prompt features, assisting the framework in better understanding the global features extracted by GFEsNet. Finally, the semantic information decoder subnetwork (SIDsNet) is introduced as a semantic information decoder, achieving efficient fusion of global spatial-spectral features and spectral prompt features, which significantly improves classification performance. Experiments conducted on four hyperspectral image datasets show that the proposed method outperforms nine existing approaches in terms of classification accuracy.