Latent Embeddings for Zero-Shot Classification

TLDR

The paper proposes a latent embedding model to learn a compatibility function between image and class embeddings for zero‑shot classification. The method augments the bilinear compatibility model by learning multiple maps and selecting one via a latent variable, trained with a ranking objective that penalizes incorrect class rankings. The model outperforms state‑of‑the‑art on three zero‑shot datasets and produces visually interpretable clusters of fine‑grained object properties.

Abstract

We present a novel latent embedding model for learning a compatibility function between image and class embeddings, in the context of zero-shot classification. The proposed method augments the state-of-the-art bilinear compatibility model by incorporating latent variables. Instead of learning a single bilinear map, it learns a collection of maps with the selection, of which map to use, being a latent variable for the current image-class pair. We train the model with a ranking based objective function which penalizes incorrect rankings of the true class for a given image. We empirically demonstrate that our model improves the state-of-the-art for various class embeddings consistently on three challenging publicly available datasets for the zero-shot setting. Moreover, our method leads to visually highly interpretable results with clear clusters of different fine-grained object properties that correspond to different latent variable maps.