On Deep Multi-View Representation Learning

Abstract

We consider learning representations (features) in the setting in which we have access to mul-tiple unlabeled views of the data for representa-tion learning while only one view is available at test time. Previous work on this problem has pro-posed several techniques based on deep neural networks, typically involving either autoencoder-like networks with a reconstruction objective or paired feedforward networks with a correlation-based objective. We analyze several techniques based on prior work, as well as new variants, and compare them experimentally on visual, speech, and language domains. To our knowledge this is the first head-to-head comparison of a vari-ety of such techniques on multiple tasks. We find an advantage for correlation-based represen-tation learning, while the best results on most tasks are obtained with our new variant, deep canonically correlated autoencoders (DCCAE). 1.