Topic Detection and Tracking using idf-Weighted Cosine Coefficient

Abstract

The goal of TDT Topic Detection and Tracking is to develop auto-matic methods of identifying topically related stories within a stream of news media. We describe approaches for both detection and track-ing based on the well-known -weighted cosine coefficient simi-larity metric. The surprising outcome of this research is that we achieved very competitive results for tracking using a very simple method of feature selection, without word stemming and without a score normalization scheme. The detection task results were not as encouraging though we attribute this more to the clustering algo-rithm than the underlying similarity metric. 1. The Tracking Task The goal of the topic tracking task for TDT2 is to identify news sto-ries on a particular event defined by a small number ( ) of positive training examples and a greater number of negative examples. All