Composing graphical models with neural networks for structured representations and fast inference

TLDR

The authors propose a general modeling and inference framework that merges probabilistic graphical models with deep learning methods. The framework composes latent graphical models with neural network observation likelihoods, uses recognition networks to generate local evidence potentials, combines them with the model distribution via efficient message‑passing, and trains all components jointly with a single stochastic variational inference objective. The framework is demonstrated by automatically segmenting and categorizing mouse behavior from raw depth video, and by illustrating several other example models.

Abstract

We propose a general modeling and inference framework that combines the complementary strengths of probabilistic graphical models and deep learning methods. Our model family composes latent graphical models with neural network observation likelihoods. For inference, we use recognition networks to produce local evidence potentials, then combine them with the model distribution using efficient message-passing algorithms. All components are trained simultaneously with a single stochastic variational inference objective. We illustrate this framework by automatically segmenting and categorizing mouse behavior from raw depth video, and demonstrate several other example models.