Tibetan Word Segmentation as Syllable Tagging Using Conditional Random Field

Abstract

Abstract. In this paper, we proposed a novel approach for Tibetan word segmentation using the conditional random field. We reformulate the segmentation as a syllable tagging problem. The approach labels each syllable with a word-internal position tag, and combines syllable(s) into words according to their tags. As there is no public available Tibetan word segmenta-tion corpus, the training corpus is generated by another segmenter which has an F-score of 96.94 % on the test set. Two feature template sets namely TMPT-6 and TMPT-10 are used and compared, and the result shows that the former is better. Experiments also show that larger training set improves the performance significantly. Trained on a set of 131,903 sentences, the segmenter achieves an F-score of 95.12 % on the test set of 1,000 sentences.