A new view of the EM algorithm that justifies incremental and other variants

Abstract

We present a new view of the EM algorithm for maximum likelihood estimation in situations with unobserved variables. In this view, both the E and the M steps of the algorithm are seen as maximizing a joint function of the model parameters and of the distribution over unobserved variables. From this perspective, it is easy to justify an incremental variant of the algorithm in which the distribution for only one of the unobserved variables is recalculated in each E step. This variant is shown empirically to give faster convergence in a mixture estimation problem. A wide range of other variant algorithms are also seen to be possible. Some key words: EM algorithm, incremental algorithms, free energy, mixtures Submitted to Biometrika 1 Introduction The Expectation-Maximization (EM) algorithm finds maximum likelihood parameter estimates in problems where some variables were unobserved. Its widespread applicability was first discussed by Dempster, Laird, and Rubin (1977). The EM algorithm ...