Cross-Modal Pattern-Propagation for RGB-T Tracking

TLDR

The authors propose a cross‑modal pattern‑propagation framework to diffuse instance patterns across RGB‑T data in both spatial and temporal domains. The method derives cross‑modal correlations from intra‑modal pattern affinities to propagate useful patterns between RGB and thermal modalities, and extends this propagation temporally by adaptively correlating long‑term historical contexts into the current frame. Extensive experiments show that CMPP achieves state‑of‑the‑art results with significant improvements on two RGB‑T tracking benchmarks.

Abstract

Motivated by our observations on RGB-T data that pattern correlations are high-frequently recurred across modalities also along sequence frames, in this paper, we propose a cross-modal pattern-propagation (CMPP) tracking framework to diffuse instance patterns across RGB-T data on spatial domain as well as temporal domain. To bridge RGB-T modalities, the cross-modal correlations on intra-modal paired pattern-affinities are derived to reveal those latent cues between heterogenous modalities. Through the correlations, the useful patterns may be mutually propagated between RGB-T modalities so as to fulfill inter-modal pattern-propagation. Further, considering the temporal continuity of sequence frames, we adopt the spirit of pattern propagation to dynamic temporal domain, in which long-term historical contexts are adaptively correlated and propagated into the current frame for more effective information inheritance. Extensive experiments demonstrate that the effectiveness of our proposed CMPP, and the new state-of-the-art results are achieved with the significant improvements on two RGB-T object tracking benchmarks.