IEEE Transactions on Multimedia · 2018 · 17 citations · 64 references
ComputingComputer ScienceComputer VisionGenerative ArtMotion PlanningAction RecognitionGenerative ModelingMotion EstimationSequence LearningVideo AnalysisEmbedding StructureGenerative Adversarial NetworksLow-dimensional EmbeddingsMotion GenerationMotion Representation LearningSynthetic Image GenerationVideo GenerationVideo Representation Learning
Video generation is a challenging task due to the extremely high-dimensional distribution of the solution space. Good constraints in the solution domain would thus reduce the difficulty of approximating optimal solutions. In this paper, instead of directly generating high-dimensional video data, we propose using object landmarks as explicit structure constraints to address this issue. Specifically, we propose a two-stage framework for an action-conditioned video generation task. In our framework, the first stage aims to generate landmark sequences according to predefined motion types, and a recurrent model (RNN/LSTM) is adopted for this purpose. The landmark sequence can be regarded as a low-dimensional structure embedding of high-dimensional video data, and generating landmark sequences is much easier than generating videos. The second stage is inspired by a conditional generative adversarial network (CGAN), and we take the generated landmark sequence as a structure condition to learn a landmark-to-image translation network. Such a one-to-one translation framework avoids the difficulty of generating videos and instead transfers the video generation task to image generation, which is resolvable due to the maturity of current GAN-based models. The experimental results demonstrate that our model not only achieves promising results on rigid/nonrigid motion generation tasks but also can be extended to multiobject motion situations.
64
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
Jun-Yan Zhu, Taesung Park, Phillip Isola et al. · 2017 · 21.9K citations · Full text