The Annals of Probability · 1988 · 178 citations · 11 references
Natural SciencesMathematicsInformation TheoryProbability TheoryStochastic ProcessesEntropyErgodic TheoryLimit TheoremsMarkov ChainsMarkov ProcessesStationary ProcessesEntropy In Ergodic TheoryEntropy RateErgodic ProcessesErgodic PropertiesIntegrability ConditionsMeasure-theoretic EntropyRelative Entropy
Let $\{X_t\}$ be a stationary ergodic process with distribution $P$ admitting densities $p(x_0,\ldots, x_{n-1})$ relative to a reference measure $M$ that is finite order Markov with stationary transition kernel. Let $I_M(P)$ denote the relative entropy rate. Then $n^{-1}\log p(X_0,\ldots, X_{n-1}) \rightarrow I_M(P) \mathrm{a.s.} (P).$ We present an elementary proof of the Shannon-McMillan-Breiman theorem and the preceding generalization, obviating the need to verify integrability conditions and also covering the case $I_M(P) = \infty$. A sandwich argument reduces the proof to direct applications of the ergodic theorem.
11
A Mathematical Theory of Communication
Claude E. Shannon · Bell System Technical Journal · 1948 · 80.7K citations
The Basic Theorems of Information Theory
Brockway McMillan · The Annals of Mathematical Statistics · 1953 · 428 citations · Full text
Large-Sample Theory: Parametric Case
Herman Chernoff · The Annals of Mathematical Statistics · 1956 · 253 citations · Full text