Visual Interaction Networks: Learning a Physics Simulator from Video

TLDR

Humans can intuitively predict the future of many physical systems from a single glance, while existing engineering, robotics, and graphics approaches are typically narrow‑domain or require explicit state information. This work introduces the Visual Interaction Network, a general‑purpose model that learns the dynamics of a physical system directly from raw visual observations. The model combines a convolutional perceptual front‑end that parses a dynamic scene into factored latent object representations with an interaction‑network dynamics predictor that rolls these states forward by computing pairwise interactions to produce arbitrary‑length trajectories. Using only six input video frames, the Visual Interaction Network accurately predicts hundreds of future time steps across diverse physical systems, can infer invisible objects and unknown masses, and enables model‑based decision‑making and planning from raw sensory data.

Abstract

From just a glance, humans can make rich predictions about the future of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains or require information about the underlying state. We introduce the Visual Interaction Network, a general-purpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.