Depth and thermal sensor fusion to enhance 3D thermographic reconstruction

TLDR

Three‑dimensional geometrical models that incorporate surface temperature data are valuable for applications such as medical imaging, energy auditing, and intelligent robotics. The study proposes a robust, mobile, real‑time 3D thermographic reconstruction method that fuses depth and thermal sensor data, leveraging temperature robustness to enable reliable scanning via a Thermal‑guided ICP approach. The method uses a calibrated multimodal device comprising a thermal camera and RGB‑D sensor, estimates initial pose by maximizing thermal consistency between infrared frames, and refines it by minimizing a combined geometric and thermographic loss function. Experiments show that fusing thermal and depth data yields more accurate 3D thermal models while requiring significantly less scanning data than conventional approaches.

Abstract

Three-dimensional geometrical models with incorporated surface temperature data provide important information for various applications such as medical imaging, energy auditing, and intelligent robots. In this paper we present a robust method for mobile and real-time 3D thermographic reconstruction through depth and thermal sensor fusion. A multimodal imaging device consisting of a thermal camera and a RGB-D sensor is calibrated geometrically and used for data capturing. Based on the underlying principle that temperature information remains robust against illumination and viewpoint changes, we present a Thermal-guided Iterative Closest Point (T-ICP) methodology to facilitate reliable 3D thermal scanning applications. The pose of sensing device is initially estimated using correspondences found through maximizing the thermal consistency between consecutive infrared images. The coarse pose estimate is further refined by finding the motion parameters that minimize a combined geometric and thermographic loss function. Experimental results demonstrate that complimentary information captured by multimodal sensors can be utilized to improve performance of 3D thermographic reconstruction. Through effective fusion of thermal and depth data, the proposed approach generates more accurate 3D thermal models using significantly less scanning data.