Learning to See Physics via Visual De-animation

TLDR

We introduce a paradigm for understanding physical scenes without human annotations. The system recovers a physical world representation with a perception module, then applies physics and graphics engines for reasoning, trained via visual de‑animation and tested by state recovery, using a convolutional inversion network to invert the engines. The system quickly recognizes physical state from appearance and motion cues, flexibly incorporates differentiable and non‑differentiable engines, and performs well on synthetic and real datasets for state estimation and reasoning, with knowledge generalizing to constrained real images.

Abstract

We introduce a paradigm for understanding physical scenes without human annotations. At the core of our system is a physical world representation that is first recovered by a perception module and then utilized by physics and graphics engines. During training, the perception module and the generative models learn by visual de-animation --- interpreting and reconstructing the visual information stream. During testing, the system first recovers the physical world state, and then uses the generative models for reasoning and future prediction. Even more so than forward simulation, inverting a physics or graphics engine is a computationally hard problem; we overcome this challenge by using a convolutional inversion network. Our system quickly recognizes the physical world state from appearance and motion cues, and has the flexibility to incorporate both differentiable and non-differentiable physics and graphics engines. We evaluate our system on both synthetic and real datasets involving multiple physical scenes, and demonstrate that our system performs well on both physical state estimation and reasoning problems. We further show that the knowledge learned on the synthetic dataset generalizes to constrained real images.