Zipf's law unzipped

Abstract

Why does Zipf's law give a good description of data from seemingly completely\nunrelated phenomena? Here it is argued that the reason is that they can all be\ndescribed as outcomes of a ubiquitous random group division: the elements can\nbe citizens of a country and the groups family names, or the elements can be\nall the words making up a novel and the groups the unique words, or the\nelements could be inhabitants and the groups the cities in a country, and so\non. A Random Group Formation (RGF) is presented from which a Bayesian estimate\nis obtained based on minimal information: it provides the best prediction for\nthe number of groups with $k$ elements, given the total number of elements,\ngroups, and the number of elements in the largest group. For each specification\nof these three values, the RGF predicts a unique group distribution\n$N(k)\\propto \\exp(-bk)/k^{\\gamma}$, where the power-law index $\\gamma$ is a\nunique function of the same three values. The universality of the result is\nmade possible by the fact that no system specific assumptions are made about\nthe mechanism responsible for the group division. The direct relation between\n$\\gamma$ and the total number of elements, groups, and the number of elements\nin the largest group, is calculated. The predictive power of the RGF model is\ndemonstrated by direct comparison with data from a variety of systems. It is\nshown that $\\gamma$ usually takes values in the interval $1\\leq\\gamma\\leq 2$\nand that the value for a given phenomena depends in a systematic way on the\ntotal size of the data set. The results are put in the context of earlier\ndiscussions on Zipf's and Gibrat's laws, $N(k)\\propto k^{-2}$ and the\nconnection between growth models and RGF is elucidated.\n