2022 IEEE International Conference on Multimedia and Expo (ICME) · 2022 · 13 citations · 20 references
ComputingComputer ScienceComputer VisionMachine LearningMachine VisionData CompressionDimension ReductionDimensionality ReductionOptimization For Machine LearningPattern RecognitionRepresentation LearningAccuracyAccuracy-computational Cost TradeoffAccuracy-pruning Trade-offData Redundancy ReductionModel CompressionPruningRedundancy ReductionTransformer Models
Vision transformers (ViT) have recently attracted considerable attentions, but the huge computational cost remains an issue for practical deployment. Previous ViT pruning methods tend to prune the model along one dimension solely, which may suffer from excessive reduction and lead to sub-optimal model quality. In contrast, we advocate a multi-dimensional ViT compression paradigm, and propose to harness the redundancy reduction from attention head, neuron and sequence dimensions jointly. We firstly propose a statistical dependence based pruning criterion that is generalizable to different dimensions for identifying deleterious components. Moreover, we cast the multi-dimensional compression as an optimization, learning the optimal pruning policy across the three dimensions that maximizes the compressed model's accuracy under a computational budget. The problem is solved by our adapted Gaussian process search with expected improvement. Experimental results show that our method effectively reduces the computational cost of various ViT models. For example, our method reduces 40% FLOPs without top-1 accuracy loss for DeiT and T2T-ViT models, outperforming previous state-of-the-arts.
20
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
Shaohui Lin, Rongrong Ji, Chenqian Yan et al. · 2019 · 565 citations