Jagiellonian University Repository (Jagiellonian University) · 2017 · 654 citations · 20 references
Open access
ComputingComputer ScienceMachine LearningMemoryStatistical LearningDeep LearningGradient-based OptimizationNeural NetworksOptimization For Machine LearningPattern RecognitionRepresentation LearningAdversarial RobustnessDropoutEffective Capacity AnalysisGradient Method OptimizationOptimization AlgorithmsRegularizationRobustness
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.
20
Multilayer feedforward networks are universal approximators
HornikK., StinchcombeM., WhiteH. · Neural Networks · 1989 · 9.4K citations
Explaining and Harnessing Adversarial Examples
Ian Goodfellow, Jonathon Shlens, Christian Szegedy · arXiv (Cornell University) · 2014 · 8.2K citations · Full text
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever et al. · arXiv (Cornell University) · 2013 · 5.7K citations · Full text
Training with Noise is Equivalent to Tikhonov Regularization
Chris Bishop · Neural Computation · 1995 · 1.3K citations