CogView: Mastering Text-to-Image Generation via Transformers

TLDR

Text‑to‑image generation remains an open challenge, demanding powerful generative models and cross‑modal understanding. The authors introduce CogView, a 4‑billion‑parameter Transformer with a VQ‑VAE tokenizer, and present finetuning strategies for downstream tasks. CogView employs a Transformer backbone with a VQ‑VAE tokenizer, and uses finetuning techniques for style learning, super‑resolution, text‑image ranking, fashion design, and pretraining stabilization such as NaN‑loss elimination. CogView attains state‑of‑the‑art FID on blurred MS COCO, surpassing prior GAN models and DALL‑E.

Abstract

Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, e.g. eliminating NaN losses. CogView achieves the state-of-the-art FID on the blurred MS COCO dataset, outperforming previous GAN-based models and a recent similar work DALL-E.