A type-token identity in the Simon-Yule model of text

Abstract

There are three significant results in this paper. First, we establish a type-token identity relating the type-token ratio and the bilogarithmic type-token ratio. The plays of Shakespeare and other interesting texts serve as demonstrative examples. Second, the Simon-Yule model of Zipf's law is used to derive the type-token identity and provide a promising statistical model of text generation. Third, a realistic refinement of the Simon-Yule model is made to allow for a decreasing entry rate of new words. Simulation methods are used to show that the type-token identity is preserved with this change in assumptions. © 1989 John Wiley & Sons, Inc.