IEEE Transactions on Circuits and Systems for Video Technology · 2024 · 12 citations · 46 references
ComputingRobotics EngineeringComputer VisionMachine VisionSensor FusionFeature-level FusionObject RecognitionRobotic VisionScene UnderstandingTracking SystemsVisual TrackingKnowledge DistillationMulti-object TrackingMultimodal Feature FusionMultimodal Feature RepresentationMultimodal FusionMultimodal Object DetectionParameter-efficient AlgorithmsResource-constrained Devices
Multi-modal object tracking has received increasing attention, given the limitations the representation ability in certain challenging scenarios of single RGB modality. Recent prompt tuning techniques enable multimodal tracking to effectively inherit knowledge from foundation models trained with a large amount of RGB tracking data and achieve parameter-efficient training. However, few works focus on the efficient inference of multimodal tracking handling multiple RGB-X (RGB-Thermal, RGB-Depth, RGB-Event, etc.) tracking tasks simultaneously, especially on resource-limited devices such as CPU. In this work, we propose an efficient multimodal tracker named EMTrack. EMTrack follows a concise and unified multimodal tracking framework with simple knowledge distillation. RGB modality and auxiliary modality are added after patch-embedding layer for fusion, reducing the computational complexity of multimodal tracking compared with that of single modality. Before fusion operation, we introduce a modal-specific spatial modulation module to exploit and realize adaptive spatial adjustment of different modality features. Multiple modal-specific experts are adopted to capture specific information for different RGB-X tracking tasks, which assists in handling such tasks in a unified model with joint training. EMTrack achieves competitive performance on various RGB-X tracking benchmarks while reaching a good balance of performance and speed on different platforms. Especially on an Intel Core i9-10850K CPU device, EMTrack achieves 29.1 fps, a real-time speed, with only 2.0G MAC computation.
46
Learning Multi-domain Convolutional Neural Networks for Visual Tracking
Hyeonseob Nam, Bohyung Han · 2016 · 2.8K citations
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking
Heng Fan, Liting Lin, Fan Yang et al. · 2019 · 1.6K citations
SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking
Dongyan Guo, Jun Wang, Ying Cui et al. · 2020 · 922 citations