Mixtures of Probabilistic Principal Component Analyzers

TLDR

Principal component analysis is widely used for data compression and visualization, but its global linearity limits performance, and although nonlinear variants exist, a probabilistic mixture of local linear PCA models has been lacking. This work formulates PCA as a maximum‑likelihood Gaussian latent variable model to enable a principled probabilistic mixture of PCA components. The resulting mixture of probabilistic principal component analyzers is defined by a Gaussian latent variable model and its parameters are learned with an expectation‑maximization algorithm. The model improves clustering, density estimation, and local dimensionality reduction, and is shown to outperform baselines on image compression and handwritten digit recognition tasks.

Abstract

Principal component analysis (PCA) is one of the most popular techniques for processing, compressing, and visualizing data, although its effectiveness is limited by its global linearity. While nonlinear variants of PCA have been proposed, an alternative paradigm is to capture data complexity by a combination of local linear PCA projections. However, conventional PCA does not correspond to a probability density, and so there is no unique way to combine PCA models. Therefore, previous attempts to formulate mixture models for PCA have been ad hoc to some extent. In this article, PCA is formulated within a maximum likelihood framework, based on a specific form of gaussian latent variable model. This leads to a well-defined mixture model for probabilistic principal component analyzers, whose parameters can be determined using an expectation-maximization algorithm. We discuss the advantages of this model in the context of clustering, density modeling, and local dimensionality reduction, and we demonstrate its application to image compression and handwritten digit recognition.