arXiv (Cornell University) · 2017 · 663 citations · 0 references
ComputingComputer ScienceFeature EngineeringMachine LearningPruning SystemsData CompressionDeep LearningDimensionality ReductionNeural NetworksOptimization For Machine LearningPruning StrategiesAccuracy-pruning Trade-offCompressionGradual PruningMemory Footprint ReductionModel CompressionPruningPruning Compression Trade-offsSparse Models
Model pruning reduces the number of nonzero parameters in deep neural networks, and recent studies show it can shrink model size by large amounts while only slightly hurting accuracy. The study questions whether over‑parameterized baseline models can be compressed simply by shrinking hidden layers instead of pruning, aiming to trade model size for accuracy. We compare large, sparsely pruned models to smaller, dense models of equal memory, and introduce a straightforward gradual pruning method that can be applied across models with minimal tuning. Across CNNs, LSTMs, and seq2seq models, large‑sparse networks consistently beat small‑dense ones, achieving up to tenfold fewer non‑zero parameters while maintaining accuracy.
Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.