2005 · 320 citations · 32 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingLanguage ModelingLanguage ModelsParsingPart-of-speech TaggingSequence LabelingWord Sense DisambiguationShallow Parsing
Conditional random fields are log‑linear models that effectively handle sequence labeling tasks such as shallow parsing and named‑entity extraction by incorporating arbitrary features. The authors aim to develop contrastive estimation, an unsupervised method for training log‑linear models on unlabeled data. Contrastive estimation learns by contrasting observed data with synthetic negative examples, exploiting implicit negative evidence to provide a computationally efficient training algorithm. Applied to POS tagging with a dictionary and unlabeled text, contrastive estimation outperforms EM, remains robust to dictionary degradation, and recovers performance by modeling additional features.
Conditional random fields (Lafferty et al., 2001) are quite effective at sequence labeling tasks like shallow parsing (Sha and Pereira, 2003) and named-entity extraction (McCallum and Li, 2003). CRFs are log-linear, allowing the incorporation of arbitrary features into the model. To train on unlabeled data, we require unsupervised estimation methods for log-linear models; few exist. We describe a novel approach, contrastive estimation. We show that the new technique can be intuitively understood as exploiting implicit negative evidence and is computationally efficient. Applied to a sequence labeling problem---POS tagging given a tagging dictionary and unlabeled text---contrastive estimation outperforms EM (with the same feature set), is more robust to degradations of the dictionary, and can largely recover by modeling additional features.
32
Unsupervised word sense disambiguation rivaling supervised methods
David Yarowsky · 1995 · 2.4K citations · Full text