Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Dong Li, Johan Björck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som,

arXiv (Cornell University) · 2022 · 151 citations · 0 references

DOIFull text

Open access

Concepts

TL;DR

A major convergence of language, vision, and multimodal pretraining is emerging. This work introduces BEiT‑3, a general‑purpose multimodal foundation model that achieves state‑of‑the‑art transfer performance on vision and vision‑language tasks. BEiT‑3 employs a Multiway Transformer backbone that supports deep fusion and modality‑specific encoding, and applies a unified masked language modeling task across images, text, and image‑text pairs. BEiT‑3 achieves state‑of‑the‑art results on a wide range of vision and vision‑language benchmarks, including COCO, ADE20K, ImageNet, NLVR2, VQAv2, and Flickr30K.

Abstract

A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).