High Fidelity Video Prediction with Large Stochastic Recurrent Neural\n Networks

Abstract

Predicting future video frames is extremely challenging, as there are many\nfactors of variation that make up the dynamics of how frames change through\ntime. Previously proposed solutions require complex inductive biases inside\nnetwork architectures with highly specialized computation, including\nsegmentation masks, optical flow, and foreground and background separation. In\nthis work, we question if such handcrafted architectures are necessary and\ninstead propose a different approach: finding minimal inductive bias for video\nprediction while maximizing network capacity. We investigate this question by\nperforming the first large-scale empirical study and demonstrate\nstate-of-the-art performance by learning large models on three different\ndatasets: one for modeling object interactions, one for modeling human motion,\nand one for modeling car driving.\n