Measuring Lexical Style and Competence: The Type-Token Vocabulary Curve

Abstract

A personal computer is used to analyze samples from literary texts by thirteen different authors, including passages from Genesis, Hemingway, Longfellow, Jane Austen, Henry James, George Eliot, James Joyce, and Basic English (created by C. K. Ogden). The total number of words (tokens) and the number of distinct vocabulary words (types) are computed for each sample. The number of types are then plotted against the number of tokens for eight of the texts. From these type-token curves, inferences are drawn about both lexical style (vocabulary use) and lexical competence (vocabulary size). For example, the curves for "Big Two-Hearted River " and for a summary of Macbeth in Basic English nearly coincide for their first 1100 tokens, after which they gradually diverge. This graphical pattern corresponds with intuitions that Hemingway's prose reads much like Basic English but that it draws upon a larger total vocabulary. The curve for Joyce's Ulysses, by contrast, rises much more rapidly than that for a late passage from A Portrait of the Artist as a Young Man; however, after 800 tokens, the two curves begin to converge. This suggests that the difference between