Data fusion through cross-modality metric learning using similarity-sensitive hashing

TLDR

Visual understanding relies on measuring similarity between observations, and learning task‑specific similarities from examples has proven beneficial across computer vision and pattern recognition, yet many applications require comparing data from disparate modalities with differing structures and dimensionalities. The paper proposes a supervised similarity learning framework that embeds data from two arbitrary spaces into Hamming space. The mapping is formulated as a binary classification problem with positive and negative examples and is efficiently learned via boosting algorithms. The approach proves useful and efficient on challenging tasks such as cross‑representation shape retrieval and multi‑modal medical image alignment.

Abstract

Visual understanding is often based on measuring similarity between observations. Learning similarities specific to a certain perception task from a set of examples has been shown advantageous in various computer vision and pattern recognition problems. In many important applications, the data that one needs to compare come from different representations or modalities, and the similarity between such data operates on objects that may have different and often incommensurable structure and dimensionality. In this paper, we propose a framework for supervised similarity learning based on embedding the input data from two arbitrary spaces into the Hamming space. The mapping is expressed as a binary classification problem with positive and negative examples, and can be efficiently learned using boosting algorithms. The utility and efficiency of such a generic approach is demonstrated on several challenging applications including cross-representation shape retrieval and alignment of multi-modal medical images.