Hierarchical Dirichlet Processes

TLDR

The paper addresses clustering problems where observations within each group are drawn from a mixture model and component sharing across groups is desired. The authors aim to develop and analyze hierarchical Dirichlet process models that enable automatic inference of shared mixture components across groups. They construct a hierarchical Dirichlet process where each group has a Dirichlet process whose base measure is itself drawn from a Dirichlet process, yielding shared atoms, and they provide stick‑breaking and Chinese‑restaurant‑franchise representations along with MCMC algorithms for posterior inference. The resulting hierarchical model guarantees that mixture components are shared across groups, and the proposed MCMC methods successfully recover these components in applications to information retrieval and text modeling.

Abstract

AbstractWe consider problems involving groups of data where each observation within a group is a draw from a mixture model and where it is desirable to share mixture components between groups. We assume that the number of mixture components is unknown a priori and is to be inferred from the data. In this setting it is natural to consider sets of Dirichlet processes, one for each group, where the well-known clustering property of the Dirichlet process provides a nonparametric prior for the number of mixture components within each group. Given our desire to tie the mixture models in the various groups, we consider a hierarchical model, specifically one in which the base measure for the child Dirichlet processes is itself distributed according to a Dirichlet process. Such a base measure being discrete, the child Dirichlet processes necessarily share atoms. Thus, as desired, the mixture models in the different groups necessarily share mixture components. We discuss representations of hierarchical Dirichlet processes in terms of a stick-breaking process, and a generalization of the Chinese restaurant process that we refer to as the “Chinese restaurant franchise.” We present Markov chain Monte Carlo algorithms for posterior inference in hierarchical Dirichlet process mixtures and describe applications to problems in information retrieval and text modeling.KEY WORDS: ClusteringHierarchical modelMarkov chain Monte CarloMixture modelNonparametric Bayesian statistics