A Probabilistic Procedure for Grouping Words into Phrases

Abstract

A procedure based on the frequency and redundancy of sequences of syntactic word-classes was devised to identify behavioural word groups, or phrases. A small sample of these phrases, derived from processing a short corpus of running text, was compared with phrases produced by immediate constituent analysis of the same text. Over 50% agreement between the two procedures was found, with a majority of the disagreements being attributable to the disparity in descriptive power between the two analytic procedures rather than to a conceptual difference in the types of word-group defined.