Collapsed variational Dirichlet process mixture models

TLDR

Nonparametric Bayesian mixture models, especially Dirichlet process mixtures, promise powerful density estimation and clustering, but their practical use on large datasets hinges on computational efficiency. The authors aim to experimentally compare several variational Bayesian approximations to the Dirichlet process mixture model. They evaluate a standard VB approach and a novel collapsed VB method—marginalizing mixture weights—using both stick‑breaking truncation and a finite symmetric Dirichlet prior.

Abstract

Nonparametric Bayesian mixture models, in particular Dirichlet process (DP) mixture models, have shown great promise for density estimation and data clustering. Given the size of today's datasets, computational efficiency becomes an essential ingredient in the applicability of these techniques to real world data. We study and experimentally compare a number of variational Bayesian (VB) approximations to the DP mixture model. In particular we consider the standard VB approximation where parameters are assumed to be independent from cluster assignment variables, and a novel collapsed VB approximation where mixture weights are marginalized out. For both VB approximations we consider two different ways to approximate the DP, by truncating the stick-breaking construction, and by using a finite mixture model with a symmetric Dirichlet prior.