2023 · 26 citations · 48 references
ComputingComputer ScienceComputer VisionMachine LearningVisual PerceptionDeep LearningLatent Variable ModelingLatent Variable ModelsRepresentation LearningSelf-supervised LearningVisual MaskingAutoencodersHierarchical Latent Variable ModelsHierarchical Representation LearningLatent Feature LearningMaskingSelf-supervised Image RepresentationsSemantic InformationVision Self-supervised Learning
Masked autoencoder (MAE), a simple and effective self-supervised learning framework based on the reconstruction of masked image regions, has recently achieved prominent success in a variety of vision tasks. Despite the emergence of intriguing empirical observations on MAE, a theoretically principled understanding is still lacking. In this work, we formally characterize and justify existing empirical in-sights and provide theoretical guarantees of MAE. We formulate the underlying data-generating process as a hierarchical latent variable model, and show that under reasonable assumptions, MAE provably identifies a set of latent variables in the hierarchical model, explaining why MAE can extract high-level information from pixels. Further, we show how key hyperparameters in MAE (the masking ratio and the patch size) determine which true latent variables to be recovered, therefore influencing the level of semantic information in the representation. Specifically, extremely large or small masking ratios inevitably lead to low-level representations. Our theory offers coherent explanations of existing empirical observations and provides insights for potential empirical improvements and fundamental limitations of the masked-reconstruction paradigm. We conduct extensive experiments to validate our theoretical insights.
48
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh et al. · IEEE Transactions on Image Processing · 2004 · 56K citations
Laurens van der Maaten, Geoffrey E. Hinton · Journal of Machine Learning Research · 2008 · 35.7K citations
Kaiming He, Georgia Gkioxari, Piotr Dollár et al. · 2017 · 28.8K citations
Feature Pyramid Networks for Object Detection
Tsung-Yi Lin, Piotr Dollár, Ross Girshick et al. · 2017 · 28.7K citations