Multi-Branch Differential Bidirectional Fusion Network for RGB-T Semantic Segmentation

Abstract

Semantic segmentation plays an important role in computer perception tasks. Integrating the rich details of RGB images with the illumination robustness of thermal infrared (TIR) images is a promising approach for achieving reliable semantic scene understanding. Current approaches for RGB-Thermal semantic segmentation often overlook the unique characteristics exhibited by each modality at different encoding layers and underutilize the complementary information between the two modalities during decoding. To acquire complementary cross-modality encoding and decoding features, we propose a multi-branch differential bidirectional fusion network known as MDBFNet. Firstly, it models the dependencies between the modality-specific characteristics and the different encoding layers, and designs a TIR-led detail enhancement module (TDE) and an RGB-led semantic enhancement module (RSE) to guide distinguishable fusion for different layer features. Secondly, a three-branch fusion decoder with three supervision (TFDS) is proposed to thoroughly explore the complementary decoding features between two modalities. Experiments on MFNet and PST900 datasets show that our method surpasses state-of-the-art methods by a clear margin.