2014 · 599 citations · 24 references
ComputingComputer ScienceComputer VisionInformation RetrievalMachine LearningDeep LearningImage RetrievalPattern RecognitionRepresentation LearningRetrieval ModelsSemantic SearchAutoencodersCanonical Correlation AnalysisCommon InformationCross-modal Representation LearningCross-modal RetrievalMultimodal Representation LearningMultimodal RetrievalMultimodal Retrieval Models
The study addresses cross‑modal retrieval, aiming to enable searching images with text queries and vice versa. They propose a correspondence autoencoder (Corr‑AE) that jointly trains two uni‑modal autoencoders by minimizing a weighted sum of reconstruction errors and a correlation loss, encouraging shared hidden representations; extensions Corr‑Cross‑AE and Corr‑Full‑AE are evaluated on three real‑scene datasets. The Corr‑AE models outperform canonical correlation analysis and two popular multimodal deep models on cross‑modal retrieval tasks.
The problem of cross-modal retrieval, e.g., using a text query to search for images and vice-versa, is considered in this paper. A novel model involving correspondence autoencoder (Corr-AE) is proposed here for solving this problem. The model is constructed by correlating hidden representations of two uni-modal autoencoders. A novel optimal objective, which minimizes a linear combination of representation learning errors for each modality and correlation learning error between hidden representations of two modalities, is used to train the model as a whole. Minimization of correlation learning error forces the model to learn hidden representations with only common information in different modalities, while minimization of representation learning error makes hidden representations are good enough to reconstruct input of each modality. A parameter $\alpha$ is used to balance the representation learning error and the correlation learning error. Based on two different multi-modal autoencoders, Corr-AE is extended to other two correspondence models, here we called Corr-Cross-AE and Corr-Full-AE. The proposed models are evaluated on three publicly available data sets from real scenes. We demonstrate that the three correspondence autoencoders perform significantly better than three canonical correlation analysis based models and two popular multi-modal deep models on cross-modal retrieval tasks.
24
Laurens van der Maaten, Geoffrey E. Hinton · Journal of Machine Learning Research · 2008 · 35.7K citations
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh · Neural Computation · 2006 · 16.4K citations