Stochastic Prediction of Multi-Agent Interactions from Partial\n Observations

Abstract

We present a method that learns to integrate temporal information, from a\nlearned dynamics model, with ambiguous visual information, from a learned\nvision model, in the context of interacting agents. Our method is based on a\ngraph-structured variational recurrent neural network (Graph-VRNN), which is\ntrained end-to-end to infer the current state of the (partially observed)\nworld, as well as to forecast future states. We show that our method\noutperforms various baselines on two sports datasets, one based on real\nbasketball trajectories, and one generated by a soccer game engine.\n