arXiv (Cornell University) · 2020 · 17 citations · 15 references
ComputingComputer ScienceArtificial IntelligenceComputer VisionMachine LearningDeep LearningNetwork InferenceNeural NetworksPruning StrategiesScalable Machine LearningAccuracy-pruning Trade-offConvolutional Neural NetworksGradual Magnitude PruningLarge-scale Data ExplorationLarge-scale Data ProcessingPruningPruning Level
We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We functionally approximate the error of the pruned networks, showing it is predictable in terms of an invariant tying width, depth, and pruning level, such that networks of vastly different pruned densities are interchangeable. We demonstrate the accuracy of this approximation over orders of magnitude in depth, width, dataset size, and density. We show that the functional form holds (generalizes) for large scale data (e.g., ImageNet) and architectures (e.g., ResNets). As neural networks become ever larger and costlier to train, our findings suggest a framework for reasoning conceptually and analytically about a standard method for unstructured pruning.
15
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Language Models are Few-Shot Learners
T. B. Brown, Benjamin Mann, Nick Ryder et al. · arXiv (Cornell University) · 2020 · 3K citations · Full text
Yann LeCun, John S. Denker, Sara A. Solla · 1989 · 2.6K citations
Russell Reed · IEEE Transactions on Neural Networks · 1993 · 1.7K citations