Marginalized Denoising Auto-encoders for Nonlinear Representations

Abstract

Denoising auto-encoders (DAEs) have been suc-cessfully used to learn new representations for a wide range of machine learning tasks. During training, DAEs make many passes over the train-ing dataset and reconstruct it from partial cor-ruption generated from a pre-specified corrupting distribution. This process learns robust represen-tation, though at the expense of requiring many training epochs, in which the data is explicitly corrupted. In this paper we present the marginal-ized Denoising Auto-encoder (mDAE), which (approximately) marginalizes out the corruption during training. Effectively, the mDAE takes into account infinitely many corrupted copies of the training data in every epoch, and therefore is able to match or outperform the DAE with much fewer training epochs. We analyze our proposed algorithm and show that it can be understood as a classic auto-encoder with a special form of reg-ularization. In empirical evaluations we show that it attains 1-2 order-of-magnitude speedup in training time over other competing approaches. 1.