arXiv (Cornell University) · 2018 · 364 citations · 20 references
ComputingComputer ScienceComputer VisionFeature EngineeringMachine LearningDeep LearningNeural NetworksOptimization For Machine LearningPruning StrategiesAccuracyConnectionsPruningPruning During Pre-trainingRecurrent Neural NetworksSaliency-based PruningSparse Network LearningTraining Pruning
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.
20
Yann LeCun, John S. Denker, Sara A. Solla · 1989 · 2.6K citations
Recurrent Neural Network Regularization
Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals · arXiv (Cornell University) · 2014 · 2.3K citations · Full text
Russell Reed · IEEE Transactions on Neural Networks · 1993 · 1.7K citations