IEEE Transactions on Information Theory · 2005 · 191 citations · 15 references
ComputingComputer ScienceData MiningMachine LearningTime Series AnalysisData PartitioningStatistical InferenceAdaptive PartitioningContinuous Probability DistributionsConvergenceDivergenceDivergence MinimizationPartitioning SchemesRadon-nikodym TheoremRegularity Criteria
We present a universal estimator of the divergence D(P/spl par/Q) for two arbitrary continuous distributions P and Q satisfying certain regularity conditions. This algorithm, which observes independent and identically distributed (i.i.d.) samples from both P and Q, is based on the estimation of the Radon-Nikodym derivative dP/dQ via a data-dependent partition of the observation space. Strong convergence of this estimator is proved with an empirically equivalent segmentation of the space. This basic estimator is further improved by adaptive partitioning schemes and by bias correction. The application of the algorithms to data with memory is also investigated. In the simulations, we compare our estimators with the direct plug-in estimator and estimators based on other partitioning approaches. Experimental results show that our methods achieve the best convergence performance in most of the tested cases.
15
On Information and Sufficiency
S. Kullback, R. A. Leibler · The Annals of Mathematical Statistics · 1951 · 20K citations · Full text
Estimation of Entropy and Mutual Information
Liam Paninski · Neural Computation · 2003 · 1.5K citations