Learning a Deep Embedding Model for Zero-Shot Learning

TLDR

Zero‑shot learning models rely on a joint embedding space that maps class semantics and visual features for nearest‑neighbour search, yet few deep end‑to‑end ZSL models exist and provide limited improvement over models that use deep features without learning such embeddings. The study argues that selecting the appropriate embedding space is essential for successful deep zero‑shot learning. The authors embed directly into the visual feature space, which reduces hubness and enables joint optimisation of multiple semantic modalities in an end‑to‑end manner. Experiments on four benchmarks demonstrate that the proposed model significantly outperforms existing zero‑shot learning approaches.

Abstract

Zero-shot learning (ZSL) models rely on learning a joint embedding space where both textual/semantic description of object classes and visual representation of object images can be projected to for nearest neighbour search. Despite the success of deep neural networks that learn an end-to-end model between text and images in other vision problems such as image captioning, very few deep ZSL model exists and they show little advantage over ZSL models that utilise deep feature representations but do not learn an end-to-end embedding. In this paper we argue that the key to make deep ZSL models succeed is to choose the right embedding space. Instead of embedding into a semantic space or an intermediate space, we propose to use the visual space as the embedding space. This is because that in this space, the subsequent nearest neighbour search would suffer much less from the hubness problem and thus become more effective. This model design also provides a natural mechanism for multiple semantic modalities (e.g.,~attributes and sentence descriptions) to be fused and optimised jointly in an end-to-end manner. Extensive experiments on four benchmarks show that our model significantly outperforms the existing models.