GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

Abstract

Generative latent-variable models are emerging as promising tools in robotics and\nreinforcement learning. Yet, even though tasks in these domains typically involve\ndistinct objects, most state-of-the-art generative models do not explicitly capture\nthe compositional nature of visual scenes. Two recent exceptions, MONet and\nIODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions.\nHence, neither of them allows for principled sampling of novel scenes. Here we\npresent GENESIS, the first object-centric generative model of rendered 3D scenes\ncapable of both decomposing and generating scenes by capturing relationships\nbetween scene components. GENESIS parameterises a spatial GMM over images\nwhich is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive\nprior. We train GENESIS on several publicly available datasets and evaluate its\nperformance on scene generation, decomposition, and semi-supervised learning.