Adaptive Iterative Pruning for Accelerating Deep Neural Networks

Abstract

The new adaptive iterative pruning (AIP) approach is proposed. It differs from the standard iterative pruning- retraining scheme by the adaptively changing hyperparameters by linear, multiplicative, or adaptive laws (batch size, number of epochs, a minimal number of channels per layer) that compensate the loss of accuracy due to excessive retraining of the survived weights. It was shown, for example of classification task by VGG-16 model on MNIST dataset, that there is some domain of values where the loss of accuracy could be not higher than 2% with the inference speedup up to ×17 and the loss of accuracy could be not higher than 4% with the inference speedup up to ×30. In the more general sense, AIP can be widely applied for any deep learning model containing the convolutional layers, i.e. for any convolutional neural networks (CNNs), with the similar speedup of inference time and potentially faster convergence during training.