Efficient Vision Transformer for Human-Centric AIoT Applications Through Token Tracking Assignment

Abstract

The integration of Artificial Intelligence Internet of Things (AIoT) with consumer electronics has resulted in enhanced connectivity and intelligence within the consumer electronics sector, thereby facilitating a more convenient living experience for individuals. Particularly, the utilization of Vision Transformer (ViT) in AIoT has further improved the decision-making capabilities of consumer electronics devices. To cater to the increasing demand for efficient and intelligent consumer electronics, the prevalent approach involves transferring ViT models, which are trained on AIoT cloud servers, to edge devices for deployment. However, the incongruity between resource-constrained edge devices and computationally intensive ViT models necessitates the reduction of redundant computations in ViT to optimize the utilization of edge devices. This paper introduces a novel approach named TMTP, which combines token merging and pruning techniques. TMTP encompasses three key blocks: the Token Merging Block (TMB), the Token Tracking Assignment Block (TAB), and the Token Pruning Block (TPB). TMTP effectively diminishes redundant computations in the ViT model while preserving its accuracy. The experimental results demonstrate that by employing TMTP to hierarchically prune 80% of the input tokens in the SWAG model, the number of GFLOPs is significantly reduced by 35%. Interestingly, the accuracy of ImageNet-1k experiences a mere reduction of 0.05%-0.18%.