On rectified linear units for speech processing

TLDR

Deep neural networks are the gold standard for acoustic modeling in speech recognition, relying on linear projections followed by point‑wise nonlinearities, typically logistic functions. The study aims to show that replacing logistic units with rectified linear units improves generalization and simplifies training. Rectified linear units output the input when positive and zero otherwise, and the networks were trained in a distributed setting on several hundred hours of speech data across hundreds of machines. The rectified linear network achieved lower word error rates than a logistic network on a large‑vocabulary task, and its learned sparse features proved useful for discriminative tasks.

Abstract

Deep neural networks have recently become the gold standard for acoustic modeling in speech recognition systems. The key computational unit of a deep network is a linear projection followed by a point-wise non-linearity, which is typically a logistic function. In this work, we show that we can improve generalization and make training of deep networks faster and simpler by substituting the logistic units with rectified linear units. These units are linear when their input is positive and zero otherwise. In a supervised setting, we can successfully train very deep nets from random initialization on a large vocabulary speech recognition task achieving lower word error rates than using a logistic network with the same topology. Similarly in an unsupervised setting, we show how we can learn sparse features that can be useful for discriminative tasks. All our experiments are executed in a distributed environment using several hundred machines and several hundred hours of speech data.