Maximum-entropy distributions having prescribed first and second moments (Corresp.)

David Dowson, A.A. Wragg

IEEE Transactions on Information Theory · 1973 · 116 citations · 2 references

Concepts

Abstract

The entropy <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</tex> of an absolutely continuous distribution with probability density function <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p(x)</tex> is defined as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H = - \int p(x) \log p(x) dx</tex> . The formal maximization of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</tex> , subject to the moment constraints <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\int x^r p(x) dx = \mu_r, r = 0,1,\cdots,m</tex> , leads to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p(x) = \exp (- \sum_{r=0}^m \lamnbda_r x^r)</tex> , where the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\lambda_r</tex> have to be chosen so as to satisfy the moment constraints. Only the case <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m = 2</tex> is considered. It is shown that when <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</tex> has finite range, a distribution maximizing the entropy exists and is unique. When the range is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[0,\infty)</tex> , the maximum-entropy distribution exists if, and only if, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\mu_2 \leq 2 \mu_1^2</tex> , and a table is given which enables the maximum-entropy distribution to be computed. The case <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\mu_2 &gt; 2 \mu_1^2</tex> is discussed in some detail.

References

2