arXiv (Cornell University) · 2012 · 5.5K citations · 6 references
ComputingComputer ScienceData ScienceAlgorithmsData MiningMachine LearningDeep LearningNeural NetworksOptimization For Machine LearningPattern RecognitionRepresentation LearningSupervised LearningAdaboostGradient DescentHyperparameter TuningLearningOptimization AlgorithmsStochastic Gradient DescentTuning
We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
6
Learning representations by back-propagating errors
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · Nature · 1986 · 30.9K citations
A Stochastic Approximation Method
Herbert Robbins, Sutton Monro · The Annals of Mathematical Statistics · 1951 · 9.6K citations · Full text
Large Scale Distributed Deep Networks
Jay B. Dean, Greg S. Corrado, Rajat Monga et al. · 2012 · 2.9K citations