Meta-Learning with Latent Embedding Optimization

TLDR

Gradient‑based meta‑learning techniques are widely applicable but struggle in high‑dimensional parameter spaces under extreme low‑data regimes. This work demonstrates that learning a data‑dependent latent generative representation of model parameters allows gradient‑based meta‑learning to be performed in a low‑dimensional latent space. The proposed latent embedding optimization (LEO) decouples the adaptation process from the high‑dimensional parameter space by optimizing in the learned latent space. LEO achieves state‑of‑the‑art performance on miniImageNet and tieredImageNet few‑shot classification, and captures data uncertainty to improve adaptation.

Abstract

Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.