IEEE Transactions on Computational Imaging · 2021 · 212 citations · 49 references
ComputingComputer ScienceComputer VisionMachine LearningMachine VisionGenerative ModelingImage Feature EngineeringImage FusionDeep Image FeaturesDeep Learning-based Image FusionGenerative Adversarial Network-based Image FusionGenerative Adversarial NetworksImage Data FusionInfrared Image FusionThermal RadiationVisible Image Fusion
A good result of infrared and visible image fusion should not only maintain significant contrast for distinguishing targets from the backgrounds, but also contain rich scene textures to cater for human visual perception. However, previous fusion methods usually do not fully utilize the information, and hence their fused results sacrifice either the salience of thermal targets or the sharpness of textures. To address this challenge, we propose a novel Generative Adversarial Network with Full-scale skip connection and dual Markovian discriminators (GAN-FM) to fully preserve effective information in infrared and visible images. First, a full-scale skip connected generator is designed to extract and fuse deep features of different scales, which can promote the direct transmission of shallow high-contrast features to the deep level, preserving the thermal radiation targets from the semantic level. As a result, the fused image can maintain significant contrast. Second, we propose two Markovian discriminators to establish adversarial games with the generator, so as to estimate probability distributions of infrared and visible modalities at the same time. Unlike conventional global discriminator, the Markovian discriminators try to distinguish each patch of input images, thus the attention of network is restricted to local regions and the fused results are forced to contain more details. In addition, we propose an effective joint gradient loss to ensure the harmonious coexistence of contrast and texture, which prevents the background texture pollution caused by the edge diffusion of the high-contrast target regions. Extensive qualitative and quantitative experiments demonstrate that our GAN-FM has advantages over the state-of-the-art methods in preserving significant contrast and rich textures. Moreover, we apply the fused image generated by our method to object detection and image segmentation, which can effectively improve the performance.
49
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Densely Connected Convolutional Networks
Gao Huang, Zhuang Liu, Laurens van der Maaten et al. · 2017 · 44.7K citations
Deep Learning for Generic Object Detection: A Survey
Li Liu, Wanli Ouyang, Xiaogang Wang et al. · International Journal of Computer Vision · 2019 · 2.8K citations · Full text
UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation
Huimin Huang, Lanfen Lin, Ruofeng Tong et al. · 2020 · 2.7K citations