Proceedings of the 30th ACM International Conference on Multimedia · 2022 · 148 citations · 14 references
ComputingComputer ScienceComputer VisionMachine VisionSensor FusionInput ModalitiesMultimodal InteractionTracking SystemsVisual TrackingBenchmark DatasetsInsufficient TrainingMulti-modal Data IntegrationMulti-modal Sensor FusionMultimodal FusionMultimodal Input Fusion
Multi‑modal tracking promises higher accuracy and robustness than RGB‑only tracking, yet it is limited by data scarcity and modality gaps that hinder effective fusion. This work proposes to address these limitations by leveraging multi‑modal visual prompts rather than constructing a fusion module. ProTrack converts multi‑modal inputs into a single modality through a prompt framework, enabling pre‑trained RGB trackers to perform high‑performance multi‑modal tracking without additional multi‑modal training. Experiments on five benchmark datasets confirm that ProTrack outperforms existing methods, validating its effectiveness.
Multi-modal tracking gains attention due to its ability to be more accurate and robust in complex scenarios compared to traditional RGB-based tracking. Its key lies in how to fuse multi-modal data and reduce the gap between modalities. However, multi-modal tracking still severely suffers from data deficiency, thus resulting in the insufficient learning of fusion modules. Instead of building such a fusion module, in this paper, we provide a new perspective on multi-modal tracking by attaching importance to the multi-modal visual prompts. We design a novel multi-modal prompt tracker (ProTrack), which can transfer the multi-modal inputs to a single modality by the prompt paradigm. By best employing the tracking ability of pre-trained RGB trackers learning at scale, our ProTrack can achieve high-performance multi-modal tracking by only altering the inputs, even without any extra training on multi-modal data. Extensive experiments on 5 benchmark datasets demonstrate the effectiveness of the proposed ProTrack.
14
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al. · arXiv (Cornell University) · 2021 · 5.3K citations · Full text
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking
Heng Fan, Liting Lin, Fan Yang et al. · 2019 · 1.6K citations
RGB-T object tracking: Benchmark and baseline
Chenglong Li, Xinyan Liang, Yijuan Lu et al. · Pattern Recognition · 2019 · 541 citations
The Seventh Visual Object Tracking VOT2019 Challenge Results
Matej Kristan, Aleš Leonardis, Michael Felsberg et al. · 2019 · 497 citations · Full text