MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking

TLDR

Computer vision has long relied on centralized benchmarks for tasks such as detection, reconstruction, and single‑object tracking, yet quantitative standards for multi‑target tracking are scarce, with the PETS dataset being the only widely used but inconsistently applied resource. This work introduces the MOTChallenge benchmark to provide a unified evaluation framework for multi‑target tracking, addressing inconsistencies in existing datasets. The authors assemble diverse datasets, curate state‑of‑the‑art tracking algorithms, and develop a standardized evaluation system to support the benchmark.

Abstract

In the recent past, the computer vision community has developed centralized benchmarks for the performance evaluation of a variety of tasks, including generic object and pedestrian detection, 3D reconstruction, optical flow, single-object short-term tracking, and stereo estimation. Despite potential pitfalls of such benchmarks, they have proved to be extremely helpful to advance the state of the art in the respective area. Interestingly, there has been rather limited work on the standardization of quantitative benchmarks for multiple target tracking. One of the few exceptions is the well-known PETS dataset, targeted primarily at surveillance applications. Despite being widely used, it is often applied inconsistently, for example involving using different subsets of the available data, different ways of training the models, or differing evaluation scripts. This paper describes our work toward a novel multiple object tracking benchmark aimed to address such issues. We discuss the challenges of creating such a framework, collecting existing and new data, gathering state-of-the-art methods to be tested on the datasets, and finally creating a unified evaluation system. With MOTChallenge we aim to pave the way toward a unified evaluation framework for a more meaningful quantification of multi-target tracking.