Convergence Theorems for Generalized Alternating Minimization Procedures

Abstract

The EM algorithm is widely used to develop iterative parameter estimation procedures for statistical models. In cases where the algorithms strictly follow the EM formulation, the convergence properties of the estimation procedures are well understood. In some instances there are practical reasons to develop procedures that do not strictly fall within the EM framework. We study EM variants in which the E-Step is not performed exactly, either to obtain improved rates of convergence, or due to approximations needed to compute statistics under a model family over which E-Steps cannot be realized. Since these variants are not EM procedures, the standard (G)EM convergence results do not apply to them. We present an information geometric framework for describing such algorithms and analysing their convergence properties. We apply this framework to analyse the convergence properties of incremental EM and variational EM. For the former, we discuss conditions under these algorithms converge in likelihood. For the latter, we characterize exactly the degree to which the E-Step approximation prevents convergence to local maxima in likelihood.