1b:I[76749,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"default"] 1c:I[3486,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"default"] 1d:I[17707,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"default"] 1e:I[66528,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"default"] 1f:I[63176,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"default"] 20:I[87099,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"default"] 21:I[26200,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"Group"] 22:I[23365,["/_next/static/chunks/bf18fea59be6d5b1.js","/_next/static/chunks/1ba21a8e081e29d1.js","/_next/static/chunks/0383663887945ae7.js","/_next/static/chunks/2060b26370dc6c0a.js","/_next/static/chunks/c89b4435b8c37fcf.js","/_next/static/chunks/8ddf2550f38c2f2a.js","/_next/static/chunks/5bf36002c846d2d8.js"],"default"] 23:T513,Vision language pre-training aims to learn alignments between vision and language from a large amount of data. Most existing methods only learn image-text alignments. Some others utilize pre-trained object detectors to leverage vision language alignments at the object level. In this paper, we propose to learn multi-grained vision language alignments by a unified pre-training framework that learns multi-grained aligning and multi-grained localization simultaneously. Based on it, we present X ²-VLM, an all-in-one model with a flexible modular architecture, in which we further unify image-text pre-training and video-text pre-training in one model. X ²-VLM is able to learn unlimited visual concepts associated with diverse text descriptions. Experiment results show that X ²-VLM performs the best on base and large scale for both image-text and video-text tasks, making a good trade-off between performance and model scale. Moreover, we show that the modular design of X ²-VLM results in high transferability for it to be utilized in any language or domain. For example, by simply replacing the text encoder with XLM-R, X ²-VLM outperforms state-of-the-art multilingual multi-modal pre-trained models without any multilingual pre-training.

Yan Zeng

Publications