arXiv (Cornell University) · 2023 · 19 citations · 0 references
Language StudiesViticultureComputer VisionMotion DesignMotion GraphicsSpeech SynthesisVideo RecordingAutoregressive ModelsLanguage Model PretrainingLarge Language ModelsTask AdaptationTask-specific AdaptationTransformer ModelsVideo GenerationZero-shot Language Modeling
The pretrained LLM serves as a foundation that can be adapted for a range of video generation tasks. We present VideoPoet, a language model capable of synthesizing high‑quality video, with matching audio, from a large variety of conditioning signals. VideoPoet uses a decoder‑only transformer that ingests multimodal inputs and is trained in two stages—pretraining with a mixture of multimodal generative objectives and task‑specific adaptation—to learn to generate video and audio. Empirical results show VideoPoet achieves state‑of‑the‑art zero‑shot video generation, producing high‑fidelity motions. Project page: http://sites.research.google/videopoet/.
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundation that can be adapted for a range of video generation tasks. We present empirical results demonstrating the model's state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions. Project page: http://sites.research.google/videopoet/