A new statistical parser based on bigram lexical dependencies

TLDR

The paper introduces a statistical parser that models dependencies between head words using probabilistic bigram relations. It extends standard bigram probability estimation to compute dependency probabilities between word pairs and employs a beam search to achieve over 200 sentences per minute with minimal accuracy loss. On Wall Street Journal data, the parser matches or exceeds SPATTER’s performance while training 40,000 sentences in under 15 minutes.

Abstract

This paper describes a new statistical parser which is based on probabilities of dependencies between head-words in the parse tree. Standard bigram probability estimation techniques are extended to calculate probabilities of dependencies between pairs of words. Tests using Wall Street Journal data show that the method performs at least as well as SPATTER (Magerman 95; Jelinek et al. 94), which has the best published results for a statistical parser on this task. The simplicity of the approach means the model trains on 40,000 sentences in under 15 minutes. With a beam search strategy parsing speed can be improved to over 200 sentences a minute with negligible loss in accuracy.