An analysis of single-layer networks in unsupervised feature learning

TLDR

Unsupervised feature learning has advanced on benchmarks such as NORB and CIFAR through increasingly complex algorithms and deep models. This study demonstrates that simple design choices—particularly the number of hidden nodes—can outweigh algorithmic sophistication or model depth in achieving high performance. The authors evaluate off‑the‑shelf methods (sparse auto‑encoders, sparse RBMs, K‑means, Gaussian mixtures) on CIFAR, NORB, and STL with single‑layer networks, systematically varying receptive field size, hidden node count, stride, and whitening. By maximizing hidden nodes and dense feature extraction, the authors attain state‑of‑the‑art accuracy (79.6 % on CIFAR‑10, 97.2 % on NORB) with a single layer, with K‑means delivering the best performance among the tested algorithms.

Abstract

A great deal of research has focused on algorithms for learning features from unlabeled data. Indeed, much progress has been made on benchmark datasets like NORB and CIFAR by employing increasingly complex unsupervised learning algorithms and deep models. In this paper, however, we show that several simple factors, such as the number of hidden nodes in the model, may be more important to achieving high performance than the learning algorithm or the depth of the model. Specifically, we will apply several othe-shelf feature learning algorithms (sparse auto-encoders, sparse RBMs, K-means clustering, and Gaussian mixtures) to CIFAR, NORB, and STL datasets using only singlelayer networks. We then present a detailed analysis of the eect of changes in the model setup: the receptive field size, number of hidden nodes (features), the step-size (“stride”) between extracted features, and the eect of whitening. Our results show that large numbers of hidden nodes and dense feature extraction are critical to achieving high performance—so critical, in fact, that when these parameters are pushed to their limits, we achieve state-of-the-art performance on both CIFAR-10 and NORB using only a single layer of features. More surprisingly, our best performance is based on K-means clustering, which is extremely fast, has no hyperparameters to tune beyond the model structure itself, and is very easy to implement. Despite the simplicity of our system, we achieve accuracy beyond all previously published results on the CIFAR-10 and NORB datasets (79.6% and 97.2% respectively).