2005 · 337 citations · 7 references
Language StudiesComputer ScienceLinguisticsComputational LinguisticsMorphologyNatural Language ProcessingMorphological AnalysisMorphological SegmentationSpeech RecognitionText SegmentationConditional Random Field SegmentationConditional Random FieldsLexical SegmentationWord Segmentation
The authors present a Chinese word segmentation system for the closed track of the Sighan Bakeoff 2005. The system employs a conditional random field sequence model that incorporates numerous linguistic features—including character identity, morphological, and character reduplication—whose morphological features are automatically extracted from training data, ensuring no bias toward any Mandarin variety. The system does not overfit to any particular Mandarin variety and attains F‑scores of 0.947 (AS), 0.943 (HK), 0.950 (PK), and 0.964 (MSR).
We present a Chinese word segmentation system submitted to the closed track of Sighan bakeoff 2005. Our segmenter was built using a conditional random field sequence model that provides a framework to use a large number of linguistic features such as character identity, morphological and character reduplication features. Because our morphological features were extracted from the training corpora automatically, our system was not biased toward any particular variety of Mandarin. Thus, our system does not overfit the variety of Mandarin most familiar to the system's designers. Our final system achieved a F-score of 0.947 (AS), 0.943 (HK), 0.950 (PK) and 0.964 (MSR).
7
The first international Chinese word segmentation Bakeoff
Richard Sproat, Thomas E. Emerson · 2003 · 213 citations · Full text
Chinese word segmentation as LMR tagging
Nianwen Xue, Libin Shen · 2003 · 158 citations · Full text