A class of Rényi information estimators for multidimensional densities

TLDR

The paper introduces a class of estimators for Rényi and Tsallis entropies of an unknown distribution in multidimensional space. The estimators are constructed from kth nearest‑neighbor distances of an i.i.d. sample of size N. They consistently estimate entropies of any order, including Shannon's, under minimal assumptions, and can be extended to estimate the statistical distance between two distributions from single samples.

Abstract

A class of estimators of the R\'{e}nyi and Tsallis entropies of an unknown distribution $f$ in $\mathbb{R}^m$ is presented. These estimators are based on the $k$th nearest-neighbor distances computed from a sample of $N$ i.i.d. vectors with distribution $f$. We show that entropies of any order $q$, including Shannon's entropy, can be estimated consistently with minimal assumptions on $f$. Moreover, we show that it is straightforward to extend the nearest-neighbor method to estimate the statistical distance between two distributions using one i.i.d. sample from each. (Wit Correction.)