arXiv (Cornell University) · 2012 · 271 citations · 72 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningOptimizationDeep LearningGradient-based OptimizationLarge-scale OptimizationNeural NetworksOptimization For Machine LearningRepresentation LearningHyperparameter OptimizationMulti-layer NetworksOptimization Algorithms
Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.
72
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Laurens van der Maaten, Geoffrey E. Hinton · Journal of Machine Learning Research · 2008 · 35.7K citations
Learning representations by back-propagating errors
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · Nature · 1986 · 30.9K citations
Leo Breiman · Machine Learning · 1996 · 17K citations · Full text
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh · Neural Computation · 2006 · 16.4K citations