Exploiting diverse knowledge sources via maximum entropy in named entity recognition

Abstract

This paper describes a novel statistical namedentity (i.e. "proper name") recognition system built around a maximum enti W framework. By working within the framework of maximum entropy. theory and utilizing a flexible object-based architecture, the system is able to make use of an extraordinarily diverse range of knowledge sources in making its tagging decisions. These knowledge sources include capitalization features, lexical features, features in- dicating the current section of text (i.e. headline or main body), and dictionaries of single or multi-wtrd terms. The purely statistical system contains no hand-generated patterns and achieves a result comparable with the best statistical systems. However, when combined with other handcoded systems, the system achieves scores that exceed the highest comparable scores thus-far published.