Estimating functions of probability distributions from a finite set of samples

Abstract

This paper addresses the problem of estimating a function of a probability distribution from a finite set of samples of that distribution. A Bayesian analysis of this problem is presented, the optimal properties of the Bayes estimators are discussed, and as an example of the formalism, closed form expressions for the Bayes estimators for the moments of the Shannon entropy function are derived. Then numerical results are presented that compare the Bayes estimator to the frequency-counts estimator for the Shannon entropy. We also present the closed form estimators, all derived elsewhere, for the mutual information, ${\mathrm{\ensuremath{\chi}}}^{2}$ covariance, and some other statistics. (c) 1995 The American Physical Society