Prompting for Multi-Modal Tracking

Jinyu Yang, Zhe Li, Feng Zheng, Aleš Leonardis, Jingkuan Song

Proceedings of the 30th ACM International Conference on Multimedia · 2022 · 148 citations · 14 references

Concepts

TL;DR

Multi‑modal tracking promises higher accuracy and robustness than RGB‑only tracking, yet it is limited by data scarcity and modality gaps that hinder effective fusion. This work proposes to address these limitations by leveraging multi‑modal visual prompts rather than constructing a fusion module. ProTrack converts multi‑modal inputs into a single modality through a prompt framework, enabling pre‑trained RGB trackers to perform high‑performance multi‑modal tracking without additional multi‑modal training. Experiments on five benchmark datasets confirm that ProTrack outperforms existing methods, validating its effectiveness.

Abstract

Multi-modal tracking gains attention due to its ability to be more accurate and robust in complex scenarios compared to traditional RGB-based tracking. Its key lies in how to fuse multi-modal data and reduce the gap between modalities. However, multi-modal tracking still severely suffers from data deficiency, thus resulting in the insufficient learning of fusion modules. Instead of building such a fusion module, in this paper, we provide a new perspective on multi-modal tracking by attaching importance to the multi-modal visual prompts. We design a novel multi-modal prompt tracker (ProTrack), which can transfer the multi-modal inputs to a single modality by the prompt paradigm. By best employing the tracking ability of pre-trained RGB trackers learning at scale, our ProTrack can achieve high-performance multi-modal tracking by only altering the inputs, even without any extra training on multi-modal data. Extensive experiments on 5 benchmark datasets demonstrate the effectiveness of the proposed ProTrack.

References

14