2024 · 113 citations · 11 references
ComputingMedia ArtsComputer VisionDigital RenderingReal-time RenderingImage EditingImage SynthesisInpaintingStylized RenderingSuper-resolutionTemporal Super-resolutionTemporal UpsamplingText-to-image GenerationVideo GenerationVideo Inpainting
Existing video models synthesize distant keyframes followed by temporal super‑resolution, which makes global temporal consistency difficult to achieve. The authors aim to create Lumiere, a text‑to‑video diffusion model that generates realistic, diverse, and coherent motion in a single pass. Lumiere employs a Space‑Time U‑Net that performs spatial and temporal down‑ and up‑sampling across multiple scales, leveraging a pre‑trained text‑to‑image diffusion model to produce full‑frame‑rate, low‑resolution videos directly. The model achieves state‑of‑the‑art text‑to‑video results and supports tasks such as image‑to‑video, video inpainting, and stylized generation.
We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.
11
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A. Efros et al. · 2018 · 12.4K citations
Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala · 2023 · 3.6K citations
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Du Tran, Heng Wang, Lorenzo Torresani et al. · 2018 · 3.5K citations