VideoFlow: A Conditional Flow-Based Model for Stochastic Video\n Generation

Abstract

Generative models that can model and predict sequences of future events can,\nin principle, learn to capture complex real-world phenomena, such as physical\ninteractions. However, a central challenge in video prediction is that the\nfuture is highly uncertain: a sequence of past observations of events can imply\nmany possible futures. Although a number of recent works have studied\nprobabilistic models that can represent uncertain futures, such models are\neither extremely expensive computationally as in the case of pixel-level\nautoregressive models, or do not directly optimize the likelihood of the data.\nTo our knowledge, our work is the first to propose multi-frame video prediction\nwith normalizing flows, which allows for direct optimization of the data\nlikelihood, and produces high-quality stochastic predictions. We describe an\napproach for modeling the latent space dynamics, and demonstrate that\nflow-based generative models offer a viable and competitive approach to\ngenerative modelling of video.\n