Infrared and visible image fusion with convolutional neural networks

Abstract

The fusion of infrared and visible images of the same scene aims to generate a composite image which can provide a more comprehensive description of the scene. In this paper, we propose an infrared and visible image fusion method based on convolutional neural networks (CNNs). In particular, a siamese convolutional network is applied to obtain a weight map which integrates the pixel activity information from two source images. This CNN-based approach can deal with two vital issues in image fusion as a whole, namely, activity level measurement and weight assignment. Considering the different imaging modalities of infrared and visible images, the merging procedure is conducted in a multi-scale manner via image pyramids and a local similarity-based strategy is adopted to adaptively adjust the fusion mode for the decomposed coefficients. Experimental results demonstrate that the proposed method can achieve state-of-the-art results in terms of both visual quality and objective assessment.