2023 · 26 citations · 5 references
HumanitiesFilm StudiesAudiovisual TranslationCinematographyFilm AestheticsFilm GenresScene ConstructionAudio RetrievalContent-based Audio RetrievalMultimodal Retrieval ModelsObject Motion AnalysisObject Motion ModelingSequence-to-sequence GenerationText-to-image GenerationVideo Generation
In this paper, we present MovieFactory, a powerful framework to generate cinematic-picture (3072x1280), film-style (multi-scene), and multi-modality (sounding) movies on the demand of natural languages. As the first fully automated movie generation model to the best of our knowledge, our approach empowers users to create captivating movies with smooth transitions using simple text inputs, surpassing existing methods that produce soundless videos limited to a single scene of modest quality. To facilitate this distinctive functionality, we leverage ChatGPT to expand user-provided text into detailed sequential scripts for movie generation. Then we bring scripts to life visually and acoustically through vision generation and audio retrieval. To generate videos, we extend the capabilities of a pretrained text-to-image diffusion model through a two-stage process. Firstly, we employ spatial finetuning to bridge the gap between the pretrained image model and the new video dataset. Subsequently, we introduce temporal learning to capture object motion. In terms of audio, we leverage sophisticated retrieval models to select and align audio elements that correspond to the plot and visual content of the movie.
5
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani et al. · 2023 · 1.9K citations
Guy Schofield, Tom Bartindale, Peter Wright · 2015 · 37 citations · Full text