IEEE Micro · 2020 · 18 citations · 52 references
ComputingComputer EngineeringComputer ScienceArtificial IntelligenceMachine LearningOptimizationData CompressionDeep LearningHardware AccelerationLossy Data CompressionOptimization For Machine LearningPattern RecognitionBayesian OptimizationCompressionModel CompressionNeural Network QuantizationOptimization AlgorithmsProgrammable SystemsPruningWeight Sparsity
Deep neural networks (DNNs) frequently contain far more weights, represented at a higher precision, than are required for the specific task, which they are trained to perform. Consequently, they can often be compressed using techniques such as weight pruning and quantization that reduce both the model size and inference time without appreciable loss in accuracy. However, finding the best compression strategy and corresponding target sparsity for a given DNN, hardware platform, and optimization objective currently requires expensive, frequently manual, trial-and-error experimentation. In this article, we introduce a programmable system for model compression called CONDENSA. Users programmatically compose simple operators, in Python, to build more complex and practically interesting compression strategies. Given a strategy and user-provided objective (such as minimization of running time), CONDENSA uses a novel Bayesian optimization-based algorithm to automatically infer desirable sparsities. Our experiments on four real-world DNNs demonstrate memory footprint and hardware runtime throughput improvements of 188× and 2.59×, respectively, using at most ten samples per search.
52
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Densely Connected Convolutional Networks
Gao Huang, Zhuang Liu, Laurens van der Maaten et al. · 2017 · 44.7K citations