A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Abstract

It is well known that enormous computational power and a mass of memory are needed in deep neural networks. That makes them difficult to apply in resource-limited environments. Therefore, many network compression and acceleration technologies have come out, among which connection pruning is widely applied due to its effectiveness and convenience. A novel connection pruning method with full model capacity on multiple sparse structures is proposed in this paper. We design a simple and efficient function called Dynamic Processing Unit (DPU) to deal with the evaluated weights. Our method has the following features: (1) Instead of being pruned directly or set to 0, the weights are controlled by the DPU to determine whether they will work for subsequent forward of the network during the iteration of pruning training. (2) It supports the traditional multi-steps prune method as well as the end-to-end training mode which can get a compressed network in only one stage where training and pruning are fused. (3) It can learn multiple useful sparse structures, including but not limited in depth-wise, filter-wise, channel-wise, 2D-filter-wise, row-wise, column-wise, connection-wise and mixed sparse structures. Our method is tested on various widely-used datasets and models, such as LeNet and ResNet on MNIST and CIFAR-10. Significantly, it achieves good performance in all these cases. Some details about our method can be found at this https://github.com/hujie369/DPU.