Sparse Filtering

TLDR

Unsupervised feature learning is effective for image, video, and audio classification, yet many existing algorithms are difficult to use and require extensive hyperparameter tuning. This work introduces sparse filtering, a simple, efficient algorithm that requires only a single hyperparameter—the number of features to learn. Unlike most methods, sparse filtering does not model the data distribution; instead it optimizes the sparsity of l2‑normalized features, a cost function that can be implemented in a few lines of MATLAB code. Sparse filtering scales gracefully to high‑dimensional inputs, learns meaningful features in additional layers via greedy layer‑wise stacking, and performs well on natural images, STL‑10 object classification, and TIMET phone classification across modalities.

Abstract

Unsupervised feature learning has been shown to be effective at learning representations that perform well on image, video and audio classification. However, many existing feature learning algorithms are hard to use and require extensive hyperparameter tuning. In this work, we present sparse filtering, a simple new algorithm which is efficient and only has one hyperparameter, the number of features to learn. In contrast to most other feature learning methods, sparse filtering does not explicitly attempt to construct a model of the data distribution. Instead, it optimizes a simple cost function – the sparsity of l2-normalized features – which can easily be implemented in a few lines of MATLAB code. Sparse filtering scales gracefully to handle high-dimensional inputs, and can also be used to learn meaningful features in additional layers with greedy layer-wise stacking. We evaluate sparse filtering on natural images, object classification (STL-10), and phone classification (TIMET), and show that our method works well on a range of different modalities.