S²ENet: Spatial–Spectral Cross-Modal Enhancement Network for Classification of Hyperspectral and LiDAR Data

Abstract

The effective utilization of multimodal data (e.g., hyperspectral and light detection and ranging (LiDAR) data) has profound implications for further development of the remote sensing (RS) field. Many studies have explored how to effectively fuse features from multiple modalities; however, few of them focus on information interactions that can effectively promote the complementary semantic content of multisource data before fusion. In this letter, we propose a spatial–spectral enhancement module (S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> EM) for cross-modal information interaction in deep neural networks. Specifically, S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> EM consists of SpAtial Enhancement Module (SAEM) for enhancing spatial representation of hyperspectral data by LiDAR features and SpEctral Enhancement Module (SEEM) for enhancing spectral representation of LiDAR data by hyperspectral features. A series of experiments and ablation studies on the Houston2013 dataset show that S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> EM can effectively facilitate the interaction and understanding between multimodal data. Our source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/likyoo/Multimodal-Remote-Sensing-Toolkit</uri> , contributing to the RS community.