A Tutorial on MM Algorithms

TLDR

Frequentist statistical problems often involve optimizing functions such as likelihoods or sums of squares, and EM algorithms—effective for maximum likelihood estimation—iteratively maximize surrogate functions without requiring missing data, illustrating that EM is a special case of the broader MM class that exploits convexity to majorize or minorize objectives. The article argues that MM algorithms should be a standard tool for statisticians and explains their principles, construction methods, attractive features, and introduces new material on constrained optimization and standard error estimation. The authors describe how to construct MM algorithms, survey existing methods, and present new approaches for constrained optimization and standard error estimation. The article provides numerous examples illustrating the discussed concepts.

Abstract

Most problems in frequentist statistics involve optimization of a function such as a likelihood or a sum of squares. EM algorithms are among the most effective algorithms for maximum likelihood estimation because they consistently drive the likelihood uphill by maximizing a simple surrogate function for the log-likelihood. Iterative optimization of a surrogate function as exemplified by an EM algorithm does not necessarily require missing data. Indeed, every EM algorithm is a special case of the more general class of MM optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. In our opinion, MM algorithms deserve to be part of the standard toolkit of professional statisticians. This article explains the principle behind MM algorithms, suggests some methods for constructing them, and discusses some of their attractive features. We include numerous examples throughout the article to illustrate the concepts described. In addition to surveying previous work on MM algorithms, this article introduces some new material on constrained optimization and standard error estimation.