Lumiere: A Space-Time Diffusion Model for Video Generation

Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, A. Stephan Antony Raj,

2024 · 113 citations · 11 references

DOIFull text

Open access

Concepts

TL;DR

Existing video models synthesize distant keyframes followed by temporal super‑resolution, which makes global temporal consistency difficult to achieve. The authors aim to create Lumiere, a text‑to‑video diffusion model that generates realistic, diverse, and coherent motion in a single pass. Lumiere employs a Space‑Time U‑Net that performs spatial and temporal down‑ and up‑sampling across multiple scales, leveraging a pre‑trained text‑to‑image diffusion model to produce full‑frame‑rate, low‑resolution videos directly. The model achieves state‑of‑the‑art text‑to‑video results and supports tasks such as image‑to‑video, video inpainting, and stylized generation.

Abstract

We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

References

11