MSTNet: A Multilevel Spectral–Spatial Transformer Network for Hyperspectral Image Classification

Abstract

Convolutional neural networks (CNN) have been widely used in hyperspectral image classification (HSIC). Although the current CNN-based methods have achieved good performance, they still face a series of challenges. For example, the receptive field is limited, information is lost in down-sampling layer, and a lot of computing resources are consumed for deep networks. To overcome these problems, we proposed a multi-level spectral-spatial transformer network (MSTNet) for HSIC. The structure of MSTNet is an image-based classification framework, which is efficient and straightforward. Based on this framework, we designed a self-attentive encoder. Firstly, HSIs are processed into sequences. Meanwhile, a learned positional embedding is added to integrate spatial information. Then, a pure transformer encoder is employed to learn feature representations. Finally, the multi-level features are processed by decoders to generate the classification results in the original image size. The experimental results based on three real hyperspectral data sets demonstrate the efficiency of the proposed method in comparison with the other related CNN-based methods.