SimMIM: a Simple Framework for Masked Image Modeling

TLDR

SimMIM is a simple masked image modeling framework that demonstrates strong representation learning by using random large‑patch masking, direct RGB regression, and a lightweight linear prediction head. SimMIM simplifies prior masked image modeling methods by eliminating block‑wise masking and complex tokenization, instead employing random large‑patch masking, direct pixel regression, and a linear prediction head, and it scales to large models using far less labeled data. On ImageNet‑1K, SimMIM attains 83.8% top‑1 accuracy with ViT‑B and 87.1% with SwinV2‑H, outperforming prior methods, and a 3B Swin V2‑G model trained with 40× less labeled data reaches state‑of‑the‑art performance on four vision benchmarks. Code is available at https://github.com/microsoft/SimMIM.

Abstract

This paper presents SimMIM, a simple framework for masked image modeling. We have simplified recently proposed relevant approaches, without the need for special designs, such as block-wise masking and tokenization via discrete VAE or clustering. To investigate what makes a masked image modeling task learn good representations, we systematically study the major components in our framework, and find that the simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a powerful pre-text task; 2) predicting RGB values of raw pixels by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K by pre-training also on this dataset, surpassing previous best approach by +0.6%. When applied to a larger model with about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to address the data-hungry issue faced by large-scale model training, that a 3B model (Swin V2-G) is successfully trained to achieve state-of-the-art accuracy on four representative vision benchmarks using 40× less labelled data than that in previous practice (JFT-3B). The code is available at https://github.com/microsoft/SimMIM.