Chinese document retrieval at TREC-6

Abstract

This report concentrates solely on the Chinese track. 2 Chinese Language In the Chinese language each character represents at least a complete syllable, rather than a letter as in other languages. Many characters are also single syllable words. The total number of characters is therefore quite large and somewhat ill-defined. A literate adult would typically recognise at least 5-6,000 characters. The various modern standards define between 10-12,000 characters, although if early and ancient literature is included the number rises to approximately 100,000. Chinese is agglutinating -- there is no space between consecutive characters, except perhaps, at the end of a sentence. Thus to perform retrieval in Chinese, the basis has to be characters unless the text is pre-segmented into words. Segmentation is difficult -- not even humans will always agree on correct segmentation, and there has been much research in successful segmentation of Chinese [1]. 3 Retrieval Task The retrieval task for the Chinese track is exactly the same as the standard ad-hoc task in TREC. A given database of texts and a fixed set of topics are supplied. The task is to return a ranked list of 1,000 documents for each of the topics. For each topic, at least one run using only the description part of the topic was encouraged. The topics were supplied in both English and Chinese. Either the English could be used so that cross lingual retrieval could be explored, or the language of the document collection could be used for monolingual experiments. A 164,811 document collection including documents from both the People's Daily and the Xinhua News Agency was used. There was no segmentation information supplied. It was 170 Megabytes as raw text. There were 26 new topics constructed. There were on average 114...