Statistical criticality arises in most informative representations

Abstract

We show that statistical criticality, i.e. the occurrence of power law\nfrequency distributions, arises in samples that are maximally informative about\nthe underlying generating process. In order to reach this conclusion, we first\nidentify the frequency with which different outcomes occur in a sample, as the\nvariable carrying useful information on the generative process. The entropy of\nthe frequency, that we call relevance, provides an upper bound to the number of\ninformative bits. This differs from the entropy of the data, that we take as a\nmeasure of resolution. Samples that maximise relevance at a given resolution -\nthat we call maximally informative samples - exhibit statistical criticality.\nIn particular, Zipf's law arises at the optimal trade-off between resolution\n(i.e. compression) and relevance. As a byproduct, we derive a bound of the\nmaximal number of parameters that can be estimated from a dataset, in the\nabsence of prior knowledge on the generative model.\n Furthermore, we relate criticality to the statistical properties of the\nrepresentation of the data generating process. We show that, as a consequence\nof the concentration property of the Asymptotic Equipartition Property,\nrepresentations that are maximally informative about the data generating\nprocess are characterised by an exponential distribution of energy levels. This\narises from a principle of minimal entropy, that is conjugate of the maximum\nentropy principle in statistical mechanics. This explains why statistical\ncriticality requires no parameter fine tuning in maximally informative samples.\n