Entropy · 2020 · 24 citations · 69 references
ComputingComputer ScienceInformation TheoryMachine LearningUncertainty QuantificationActive LearningBayesian FilteringBayesian InferenceBayesian LearningBayesian ModelingBayesian StatisticsActive Learning With Gaussian ProcessesBayesian Active LearningGaussian Process EmulationGaussian Process SurrogatesInformation EntropyInformation GainModel EvidenceRelative Entropy
Gaussian process emulators (GPE) are a machine learning approach that replicates computational demanding models using training runs of that model. Constructing such a surrogate is very challenging and, in the context of Bayesian inference, the training runs should be well invested. The current paper offers a fully Bayesian view on GPEs for Bayesian inference accompanied by Bayesian active learning (BAL). We introduce three BAL strategies that adaptively identify training sets for the GPE using information-theoretic arguments. The first strategy relies on Bayesian model evidence that indicates the GPE's quality of matching the measurement data, the second strategy is based on relative entropy that indicates the relative information gain for the GPE, and the third is founded on information entropy that indicates the missing information in the GPE. We illustrate the performance of our three strategies using analytical- and carbon-dioxide benchmarks. The paper shows evidence of convergence against a reference solution and demonstrates quantification of post-calibration uncertainty by comparing the introduced three strategies. We conclude that Bayesian model evidence-based and relative entropy-based strategies outperform the entropy-based strategy because the latter can be misleading during the BAL. The relative entropy-based strategy demonstrates superior performance to the Bayesian model evidence-based strategy.
69
A Mathematical Theory of Communication
Claude E. Shannon · Bell System Technical Journal · 1948 · 80.7K citations
On Information and Sufficiency
S. Kullback, R. A. Leibler · The Annals of Mathematical Statistics · 1951 · 20K citations · Full text
Robert E. Kass, Adrian E. Raftery · Journal of the American Statistical Association · 1995 · 12.3K citations
Natural Sciences, Behavioral Sciences, Bayesian Biostatistics +11
Norbert Wiener · American Journal of Mathematics · 1938 · 3K citations