Automatic differentiation in machine learning: a survey

TLDR

Derivatives, especially gradients and Hessians, are ubiquitous in machine learning, yet automatic differentiation—an efficient, general technique for evaluating derivatives of numeric functions—has only recently begun to permeate the field through dynamic computational graphs and differentiable programming. This survey examines the intersection of automatic differentiation and machine learning, catalogues relevant applications, and discusses key implementation techniques. The authors clarify terminology by precisely defining the main techniques and their interrelationships, thereby improving understanding of autodiff, automatic differentiation, and symbolic differentiation in machine learning contexts.

Abstract

Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic (AD), also called algorithmic or simply auto-diff, is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational uid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names dynamic computational graphs and differentiable programming. We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main techniques and their interrelationships, we aim to bring clarity to the usage of the terms autodiff, automatic differentiation, and symbolic differentiation as these are encountered more and more in machine learning settings.