Shallow parsing with conditional random fields

TLDR

Conditional random fields improve sequence labeling by outperforming generative models and per‑position classifiers, and shallow parsing has become a benchmark task with extensive datasets and method comparisons. The study demonstrates training a conditional random field that matches or surpasses the best base noun‑phrase chunking methods on the CoNLL shallow parsing task. Modern optimization techniques for training CRFs yield performance that matches or exceeds prior best models, and extensive comparisons validate these improvements on shallow parsing.

Abstract

Conditional random fields for sequence labeling offer advantages over both generative models like HMMs and classifiers applied at each sequence position. Among sequence labeling tasks in language processing, shallow parsing has received much attention, with the development of standard evaluation datasets and extensive comparison among methods. We show here how to train a conditional random field to achieve performance as good as any reported base noun-phrase chunking method on the CoNLL task, and better than any reported single model. Improved training methods based on modern optimization algorithms were critical in achieving these results. We present extensive comparisons between models and training methods that confirm and strengthen previous results on shallow parsing and training methods for maximum-entropy models.