CoCa: Contrastive Captioners are Image-Text Foundation Models

TLDR

Large‑scale pretrained foundation models are highly sought after in computer vision because they can be rapidly transferred to many downstream tasks. This work introduces Contrastive Captioner (CoCa), a minimalist image‑text encoder‑decoder model trained jointly with contrastive and captioning losses to combine the strengths of CLIP‑style contrastive learning and SimVLM‑style generation. CoCa removes cross‑attention from the first half of its decoder layers to encode unimodal text, then cascades the remaining layers to cross‑attend the image encoder for multimodal representations, applying a contrastive loss between unimodal image and text embeddings and a captioning loss on the multimodal decoder outputs, all within a shared computational graph trained end‑to‑end on web‑scale alt‑text and annotated images treating labels as text. The model attains state‑of‑the‑art zero‑shot and fine‑tuned performance across a wide range of tasks, achieving 86.3 % zero‑shot ImageNet top‑1 accuracy, 90.6 % with a frozen encoder and learned head, and 91.0 % with a finetuned encoder, while also outperforming prior methods on retrieval, VQA, and captioning benchmarks.

Abstract

Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.