Zhongwen xinxi xuebao · 2002 · 40 citations · 0 references
ComputingComputer ScienceComputational LinguisticsInformation RetrievalNatural Language ProcessingLanguage ModelingText SegmentationLexical SegmentationShortest PathsStatistical Text SegmentationWord Segmentation
Rough segmentation, the initial step of Chinese word segmentation, aims to cover the correct segmentation with as few candidates as possible. This study proposes a rough segmentation model based on the N‑shortest‑paths method to achieve that goal. The model constructs a word graph, attaches frequency‑based edge weights to form a statistical model, and is evaluated on a one‑month People’s Daily news corpus of 185,192 sentences. Using the 2‑shortest‑paths non‑statistical model yields a 99.73 % recall, while the statistical model reaches 99.94 % recall—about 6.4 % higher than the best known approach and 15 % higher than maximum matching—using only 6.12 candidates on average and reducing candidate numbers by 64‑fold.
As the very first step of Chinese word segmentation,rough segmentation tries to cover the correct segmentation with as few candidates as possible. This paper presents a model of rough segmentation, which is based on the N-shortest-paths method,to achieve the goal. In parallel,a statistical model can easily be obtained by attaching frequencies to the edges of the word-graphs. Experiments have been made on a one-month news corpus of 185,192 sentences from the People s Daily. By sentence,the recalling rate of the non-statistical model based on 2-shortest-paths method is 99.73 % . When the statistical model is applied, a recalling rate as high as 99. 94 % , nearly 6.4% higher than known best approach and 15% higher than the maximum matching segmentation, can be reached with 6.12 candidates on average. In addition, the average number of segmentation candidates is reduced by 64 times as compared to the approach of full segmentation. The result shows that the N-shortest-paths method is effective for the task of rough segmentation.