IEEE/CAA Journal of Automatica Sinica · 2024 · 47 citations · 58 references
ComputingComputer ScienceComputer VisionMachine VisionSensor FusionDecision-level FusionFeature-level FusionImage FusionNear-infrared ImagingObject RecognitionScene UnderstandingCross-modal FusionImage Data FusionInfrared Image FusionMultimodal FusionVisible Image Fusion
The goal of infrared and visible image fusion (IVIF) is to integrate the unique advantages of both modalities to achieve a more comprehensive understanding of a scene. However, existing methods struggle to effectively handle modal disparities, resulting in visual degradation of the details and prominent targets of the fused images. To address these challenges, we introduce PromptFusion, a prompt-based approach that harmoniously combines multi-modality images under the guidance of semantic prompts. Firstly, to better characterize the features of different modalities, a contourlet autoencoder is designed to separate and extract the high-/low-frequency components of different modalities, thereby improving the extraction of fine details and textures. We also introduce a prompt learning mechanism using positive and negative prompts, leveraging Vision-Language Models to improve the fusion model's understanding and identification of targets in multi-modality images, leading to improved performance in downstream tasks. Furthermore, we employ bi-level asymptotic convergence optimization. This approach simplifies the intricate non-singleton non-convex bi-level problem into a series of convergent and differentiable single optimization problems that can be effectively resolved through gradient descent. Our approach advances the state-of-the-art, delivering superior fusion quality and boosting the performance of related downstream tasks. Project page: https://github.com/hey-it-s-me/PromptFusion.
58
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Learning to Prompt for Vision-Language Models
Kaiyang Zhou, Jingkang Yang, Chen Change Loy et al. · International Journal of Computer Vision · 2022 · 2.7K citations · Full text
FusionGAN: A generative adversarial network for infrared and visible image fusion
Jiayi Ma, Wei Yu, Pengwei Liang et al. · Information Fusion · 2018 · 2.1K citations