Evaluation campaigns and TRECVid

TLDR

TRECVid is an international benchmarking activity for video information retrieval that provides a large test collection, uniform scoring procedures, and a forum for nearly 70 research organizations, universities, and consortia, and it has benchmarked interactive and automatic search, feature detection, shot boundary, and story boundary detection since its inception. The paper introduces information retrieval evaluation from both user and system perspectives, emphasizing that system evaluation dominates the field, and uses TRECVid as a case study. It summarizes TRECVid as a system‑evaluation benchmarking campaign to examine whether such campaigns are beneficial or detrimental. The authors conclude that, despite arguments for and against, these campaigns have overall had a very positive impact on research progress.

Abstract

The TREC Video Retrieval Evaluation (TRECVid)is an international benchmarking activity to encourage research in video information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations 1 interested in comparing their results. TRECVid completed its fifth annual cycle at the end of 2005 and in 2006 TRECVid will involve almost 70 research organizations, universities and other consortia. Throughout its existence, TRECVid has benchmarked both interactive and automatic/manual searching for shots from within a video corpus,automatic detection of a variety of semantic and low-level video features, shot boundary detection and the detection of story boundaries in broadcast TV news. This paper will give an introduction to information retrieval (IR) evaluation from both a user and a system perspective, high-lighting that system evaluation is by far the most prevalent type of evaluation carried out. We also include a summary of TRECVid as an example of a system evaluation bench-marking campaign and this allows us to discuss whether such campaigns are a good thing or a bad thing. There are arguments for and against these campaigns and we present some of them in the paper concluding that on balance they have had a very positive impact on research progress.