2020 · 109 citations · 41 references
ComputingVision ScienceComputer VisionReal-time RenderingVisual PerceptionAction RecognitionFacial ExpressionImage FusionImage SynthesisConvolutionConvolutional Neural NetworksFacial Expression AnalysisFacial Expression SynthesisFiltersNeural Video GenerationSpatio-temporal RepresentationVideo Generation
Generating human videos from a single image requires simultaneously realistic appearance and motion. The paper proposes ImaGINator, a conditional GAN that decomposes appearance and motion to generate realistic videos from a single image and a condition. ImaGINator uses a spatio‑temporal fusion scheme that preserves appearance across layers and a factorized transposed (1+2)D convolution for efficient, high‑quality video synthesis, evaluated on MUG, UvA‑NEMO, NATOPS, and Weizmann. The approach achieves significantly better quantitative and qualitative results than the state‑of‑the‑art. Source code and models are available at https://github.com/wyhsirius/ImaGINator.
Generating human videos based on single images entails the challenging simultaneous generation of realistic and visual appealing appearance and motion. In this context, we propose a novel conditional GAN architecture, namely ImaGINator, which given a single image, a condition (label of a facial expression or action) and noise, decomposes appearance and motion in both latent and high level feature spaces, generating realistic videos. This is achieved by (i) a novel spatio-temporal fusion scheme, which generates dynamic motion, while retaining appearance throughout the full video sequence by transmitting appearance (originating from the single image) through all layers of the network. In addition, we propose (ii) a novel transposed (1+2)D convolution, factorizing the transposed 3D convolutional filters into separate transposed temporal and spatial components, which yields significantly gains in video quality and speed. We extensively evaluate our approach on the facial expression datasets MUG and UvA-NEMO, as well as on the action datasets NATOPS and Weizmann. We show that our approach achieves significantly better quantitative and qualitative results than the state-of-the-art. The source code and models are available under https://github.com/wyhsirius/ImaGINator.
41
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, Trevor Darrell · 2015 · 36.9K citations
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
Jun-Yan Zhu, Taesung Park, Phillip Isola et al. · 2017 · 21.9K citations · Full text
Auto-Encoding Variational Bayes
Diederik P. Kingma, Max Welling · UvA-DARE (University of Amsterdam) · 2013 · 15.6K citations · Full text