2006 · 58 citations · 5 references
ComputingComputer ScienceComputational LinguisticsInformation RetrievalNatural Language ProcessingLanguage ModelingLanguage ModelsPart-of-speech TaggingText SegmentationBilingual CorporaConditional Random FieldsLexical SegmentationSubword ModelingWord Segmentation
We proposed two approaches to improve Chinese word segmentation: a subword-based tagging and a confidence measure approach. We found the former achieved better performance than the existing character-based tagging, and the latter improved segmentation further by combining the former with a dictionary-based segmentation. In addition, the latter can be used to balance out-of-vocabulary rates and in-vocabulary rates. By these techniques we achieved higher F-scores in CITYU, PKU and MSR corpora than the best results from Sighan Bakeoff 2005.
5
A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005
Huihsin Tseng, Pi-Chuan Chang, Galen Andrew et al. · 2005 · 337 citations
The Second International Chinese Word Segmentation Bakeoff.
Thomas E. Emerson · 2005 · 308 citations
Chinese word segmentation as LMR tagging
Nianwen Xue, Libin Shen · 2003 · 158 citations · Full text