A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms

TLDR

The relation between the Monte Carlo EM (MCEM) algorithm and the data augmentation algorithm is noted. The article presents a Monte Carlo implementation of the EM algorithm’s E step and introduces two modifications that enable full posterior calculation. Latent data patterns are generated from the conditional predictive distribution, the augmented log‑posterior is updated as a mixture over these patterns, and the mixture is maximized in the M step to update the posterior maximizer, with gradient and Hessian also expressed as mixtures. The approximations serve as diagnostics for the normal approximation’s validity and as starting points for full data augmentation, illustrated with two examples. Key Words: Bayesian inference, Multiple imputation, Simulation.

Abstract

Abstract The first part of this article presents the Monte Carlo implementation of the E step of the EM algorithm. Given the current guess to the maximizer of the posterior distribution, latent data patterns are generated from the conditional predictive distribution. The expected value of the augmented log-posterior is then updated as a mixture of augmented log-posteriors, mixed over the generated latent data patterns (multiple imputations). In the M step of the algorithm, this mixture is maximized to obtain the update to the maximizer of the observed posterior. The gradient and Hessian of the observed log posterior are also expressed as mixtures, mixed over the multiple imputations. The relation between the Monte Carlo EM (MCEM) algorithm and the data augmentation algorithm is noted. Two modifications to the MCEM algorithm (the poor man's data augmentation algorithms), which allow for the calculation of the entire posterior, are then presented. These approximations serve as diagnostics for the validity of the normal approximation to the posterior, as well as starting points for the full data augmentation analysis. The methodology is illustrated with two examples. Key Words: Bayesian inferenceMultiple imputationSimulation