SIMULATING RATIOS OF NORMALIZING CONSTANTS VIA A SIMPLE IDENTITY: A THEORETICAL EXPLORATION

TLDR

Estimating ratios of normalizing constants for two densities, common in Bayesian inference and fields such as physics and genetics, is often tackled via identities such as \(c_1c_2=E_2(q_1\alpha)E_1(q_2\alpha)\) derived from MCMC samples. The paper theoretically investigates the identity’s usefulness, aiming to identify optimal or practical \(\alpha\) choices and to generalize it for more complex ratio‑estimation settings. The method relies on the identity \(c_1c_2=E_2(q_1\alpha)E_1(q_2\alpha)\), where expectations are taken under the two densities and \(\alpha\) is any function ensuring a non‑zero denominator. With sensible \(\alpha\) choices, the identity can reduce simulation error by orders of magnitude compared to conventional importance sampling (\(\alpha=1/q_2\)).

Abstract

Let pi(w) ,i =1 , 2, be two densities with common support where each density is known up to a normalizing constant: pi(w )= qi(w)/ci .W e have draws from each density (e.g., via Markov chain Monte Carlo), and we want to use these draws to simulate the ratio of the normalizing constants, c1/c2. Such a compu- tational problem is often encountered in likelihood and Bayesian inference, and arises in fields such as physics and genetics. Many methods proposed in statistical and other literature (e.g., computational physics) for dealing with this problem are based on various special cases of the following simple identity: c1 c2 = E2(q1(w)α(w)) E1(q2(w)α(w)) . Here Ei denotes the expectation with respect to pi (i =1 , 2), and α is an arbitrary function such that the denominator is non-zero. A main purpose of this paper is to provide a theoretical study of the usefulness of this identity, with focus on (asymptotically) optimal and practical choices of α. Using a simple but informa- tive example, we demonstrate that with sensible (not necessarily optimal) choices of α, we can reduce the simulation error by orders of magnitude when compared to the conventional importance sampling method, which corresponds to α =1 /q2. We also introduce several generalizations of this identity for handling more compli- cated settings (e.g., estimating several ratios simultaneously) and pose several open problems that appear to have practical as well as theoretical value. Furthermore, we discuss related theoretical and empirical work.