IEEE Transactions on Geoscience and Remote Sensing · 2021 · 79 citations · 41 references
ComputingEarth SciencesComputer VisionRemote SensingSensor FusionFeature-level FusionImage FusionLidar Remote SensingRemote Sensing Image AnalysisRemote Sensing ImageryRemote Sensing IntegrationAdaptive FusionFeature FusionFusion Of Heterogeneous SensorsMultimodal Data FusionMultimodal FusionMultisensor Feature FusionMultisensor FusionRemote Sensing Data Fusion
Extracting semantic information from very-high-resolution (VHR) aerial images is a prominent topic in the Earth observation research. An increasing number of different sensor platforms are appearing in remote sensing, each of which can provide corresponding multimodal supplemental or enhanced information, such as optical images, light detection and ranging (LiDAR) point clouds, infrared images, or inertial measurement unit (IMU) data. However, these current deep networks for LiDAR and VHR images have not fully utilized the complete potential of multimodal data. The stacked multimodal fusion network (MFNet) ignores the structural differences between the modalities and the manual statistical characteristics within the modalities. For multimodal remote sensing data and its corresponding carefully designed handcrafted features, we designed a novel deep MFNet that can use multimodal VHR aerial images and LiDAR data and the corresponding intramodal features, such as LiDAR-derived features [slope and normalized digital surface model (NDSM)] and imagery-derived features [infrared–red–green (IRRG), normalized difference vegetation index (NDVI), and difference of Gaussian (DoG)]. Technically, we introduce the attention mechanism and multimodal learning to adaptively fuse intermodal and intramodal features. Specifically, we designed a multimodal fusion mechanism, pyramid dilation blocks, and a multilevel feature fusion module. Through these modules, our network realized the adaptive fusion of multimodal features, improved the receptive field, and enhanced the global-to-local contextual fusion effect. Moreover, we used a multiscale supervision training scheme to optimize the network. Extensive experimental results and ablation studies on the ISPRS semantic dataset and IEEE GRSS DFC Zeebrugge dataset show the effectiveness of our proposed MFNet.
41
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, Trevor Darrell · 2015 · 36.9K citations
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2015 · 18.8K citations
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi et al. · 2017 · 15.5K citations
Squeeze-and-Excitation Networks
Jie Hu, Li Shen, Samuel Albanie et al. · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2019 · 12.5K citations