A Tutorial on Energy-Based Learning

TLDR

Energy‑Based Models assign a scalar energy to each variable configuration, offering a unified framework for diverse learning methods while avoiding the normalization challenges of probabilistic models. Inference clamps observed variables and seeks energy‑minimizing configurations, while learning adjusts the energy function so that observed states receive lower energies than unobserved ones, eliminating the need for normalization. EBMs can be viewed as non‑probabilistic factor graphs, providing greater flexibility in architecture and training design than probabilistic approaches.

Abstract

Energy-Based Models (EBMs) capture dependencies between variables by associating a scalar energy to each configuration of the variab les. Inference consists in clamping the value of observed variables and finding config urations of the remaining variables that minimize the energy. Learning consists in finding an energy function in which observed configurations of the variables a re given lower energies than unobserved ones. The EBM approach provides a common theoretical framework for many learning models, including traditional discr iminative and generative approaches, as well as graph-transformer networks, co nditional random fields, maximum margin Markov networks, and several manifold learning methods. Probabilistic models must be properly normalized, which sometimes requires evaluating intractable integrals over the space of all poss ible variable configurations. Since EBMs have no requirement for proper normalization, this problem is naturally circumvented. EBMs can be viewed as a form of non-probabilistic factor graphs, and they provide considerably more flexibility in th e design of architectures and training criteria than probabilistic approaches .