Semantic Coherence Guided Multiview Similarity for Image Classification With Varied Supervision

Abstract

Pair-wise similarity has been widely used for image classification by propagating the class information from labeled images to unlabeled images and predicting the classes of unlabeled images accordingly. Although widely used, pair-wise similarity based classification methods have two drawbacks. On one hand, visual-semantic similarity consistency does not always hold due to semantic gap. On the other hand, the reliability of pair-wise similarity is also influenced by the number and underlying distribution of labeled images. A well designed classification model cannot be easily generalized to few-shot classification tasks. To solve these two problems, in this paper, we propose a semantic coherence guided multi-view similarity (SCG-MV) for image classification method with varied supervision levels. We measure the semantic coherence between image-pairs for classifier learning. This is achieved by first conducting line search along the line segment between each image pair, and then measure the semantics of each interval. The similarity of one image-pair can then be obtained by aggregating the semantic coherence of all the intervals. In this way, we can model image similarities more finely and discriminatively. To make use of multi-view information, we also use the semantic coherence guided image similarities to learn multi-view classifiers. The proposed semantic coherence guided method can be combined with other similarity based methods for image classification with various levels of supervision. We conduct both few-shot and fully-supervised classification experiments on four datasets with the results well prove the discriminative power and generalization ability of the proposed method.