Multimodal Image Fusion via Self-Supervised Transformer

Abstract

Multi-modal image fusion (MMIF) aims to synthesize one more informative image from complementary multi-modal images. Due to the absence of ground truth, most MMIF works utilize unsupervised loss functions to preserve specific source information into the fused image, suffering from subjective information preservation and high demand for a large training set. Self-supervised learning is a feasible technology to avoid these issues. Current self-supervised MMIF works pretrain the network using a large natural image set via the vanilla image reconstruction task, then implement image fusion by integrating fusion rules into the well-trained network. However, they ignore the domain discrepancy between training data (i.e., the natural image set) and test data (i.e., the MMIF set). In addition, existing MMIF methods are mostly built based on traditional convolutional neural networks, leading to the inability of modelling long-range contextual relationships of source modalities. To this end, we propose a self-supervised transformer for multi-modal image fusion. Under the self-supervised framework, we design random image super-resolution (RSR) as the pretext task to train the network. Both RSR and MMIF have similar multi-resolution inputs and common goals: image restoration and enhancement, which are beneficial for diminishing the domain discrepancy between the train and test data. Moreover, we present a transformer-based fusion network for capturing both local and global information of source modalities. The proposed method is validated on multiple MMIF datasets. Qualitative and quantitative experimental results demonstrate that the proposed method can yield superior fusion quality compared to the state-of-the-art image fusion methods.