Transformer-Based End-to-End Anatomical and Functional Image Fusion

Abstract

Medical image fusion aims to derive complementary information from medical images with different modalities and is becoming increasingly important in clinical applications. The design of fusion strategy plays a key role in achieving high-quality fusion results. Existing methods usually employ handcrafted fusion rules or convolution-based networks to fuse multi-modal medical images. However, these fusion strategies are insufficiently fine-grained and cannot capture global information of multi-modal images effectively. Moreover, for deep learning-based fusion methods, they always concatenate source images/deep features of different modalities to input into the neural network, easily leading to inadequate information utilization from source images. To address these problems, we propose a transformer-based end-to-end framework for medical image fusion, termed as TransFusion. The proposed TransFusion introduces the transformer as the fusion strategy, which utilizes its self-attention mechanism to incorporate the global contextual information of multi-modal features and fuse them adequately. Besides, unlike traditional parallel multi-branch architectures or shared networks used for multiple inputs, we design branch networks that interact through fusion transformers at multiple scales to utilize the information of different modalities more adequately. A natural advantage of our design is the ability to aggregate global multi-modal features through self-attention. Both qualitative and quantitative experiments demonstrate the superiority of our method over the state-of-the-art fusion methods.