Neural Information Processing Systems · 2013 · 431 citations · 5 references
ComputingComputer ScienceMathematicsMachine LearningStatistical LearningStochastic ProcessesDeep LearningLoss FunctionNeural NetworksPattern RecognitionRepresentation LearningSupervised LearningConvolutional Neural NetworksDropoutGradient DescentRegularized LossStochastic Gradient DescentWeighted Geometric Mean
Dropout is a relatively new algorithm for training neural networks which relies on stochastically dropping out neurons during training in order to avoid the co-adaptation of feature detectors. We introduce a general formalism for studying dropout on either units or connections, with arbitrary probability values, and use it to analyze the averaging and regularizing properties of dropout in both linear and non-linear networks. For deep neural networks, the averaging properties of dropout are characterized by three recursive equations, including the approximation of expectations by normalized weighted geometric means. We provide estimates and bounds for these approximations and corroborate the results with simulations. Among other results, we also show how dropout performs stochastic gradient descent on a regularized error function.
5
The dropout learning algorithm
Pierre Baldi, Peter Sadowski · Artificial Intelligence · 2014 · 351 citations · Full text
Improving Neural Networks with Dropout
Nitish Srivastava · 2013 · 271 citations