Neurocomputing · 2022 · 97 citations · 29 references
ComputingComputer ScienceComputer VisionFeature EngineeringMachine LearningData CompressionDeep LearningLossless Data CompressionLossy Data CompressionNeural NetworksCompressionCompression TechniquesConvolutional Neural NetworksEnergy ConsumptionFully Convolutional NetworksModel CompressionNeural Data CompressionStructure-preserving Model Reduction
The state-of-the-art performance for several real-world problems is currently reached by deep and, in particular, convolutional neural networks (CNN). Such learning models exploit recent results in the field of deep learning, leading to highly performing, yet very large neural networks with typically millions to billions of parameters. As a result, such models are often redundant and excessively oversized, with a detrimental effect on the environment in terms of unnecessary energy consumption and a limitation to their deployment on low-resource devices. The necessity for compression techniques able to reduce the number of model parameters and their resource demand is thereby increasingly felt by the research community. In this paper we propose the first extensive comparison, to the best of our knowledge, of the main lossy and structure-preserving approaches to compress pre-trained CNNs, applicable in principle to any existing model. Our study is intended to provide a first and preliminary guidance to choose the most suitable compression technique when there is the need to reduce the occupancy of pre-trained models. Both convolutional and fully-connected layers are included in the analysis. Our experiments involved two pre-trained state-of-the-art CNNs (proposed to solve classification or regression problems) and five benchmarks, and gave rise to important insights about the applicability and performance of such techniques w.r.t. the type of layer to be compressed and the category of problem tackled.
29
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Mark Sandler, Andrew Howard, Menglong Zhu et al. · 2018 · 25.4K citations
Sinno Jialin Pan, Qiang Yang · IEEE Transactions on Knowledge and Data Engineering · 2009 · 23.2K citations
Channel Pruning for Accelerating Very Deep Neural Networks
Yihui He, Xiangyu Zhang, Jian Sun · 2017 · 2.5K citations
Comprehensive analysis of kinase inhibitor selectivity
Mindy I. Davis, Jeremy P. Hunt, Sanna Herrgård et al. · Nature Biotechnology · 2011 · 2.4K citations · Full text