A Probabilistic Parser and Its Application

Abstract

We describe a general approach to the probabilistic parsing of context-free grammars. The method integrates context-sensitive statistical knowledge of various types (e.g., syntactic and semantic) and can be trained incrementally from a bracketed corpus. We introduce a variant of the GHR contextfree recognition algorithm, and explain how to adapt it for efficient probabilistic parsing. On a real-world corpus of sentences from software testing documents, with 23 possible parses for a sentence of average length, the system accurately finds the correct parse in 99% of cases, while producing only 1.02 parses per sentence. Significantly, the success rate would be only 66% without the semantic statistics. Introduction In constrained domains, natural language processing can often provide leverage. At AT&T, for instance, NL technology can potentially help automate many aspects of software development. A typical example occurs in the software testing area. Here 250,000 English sentences specif...