2023 · 18 citations · 10 references
ComputingComputer EngineeringComputer ScienceArtificial IntelligenceComputer VisionReconfigurable ComputingAccelerator ArchitectureAttention NetworksDeep LearningFpga ArchitectureHardware AccelerationLow-power Fpga DesignAcceleratorsAttentionComputational EfficiencyConvolutionConvolutional Neural NetworksDeep Learning AcceleratorsEnergy-efficient Hardware AccelerationPower-efficient Accelerator Design
Many models combining Transformers with convolutional neural networks (CNNs) for computer vision tasks have achieved state-of-the-art results. However, due to the different computation patterns between attention and convolution, using a dedicated Transformer or CNN accelerator will inevitably reduce the computing efficiency of the other. To overcome this problem, we propose a unified architecture for attention and convolution on FPGA. We reduce runtime overhead by offloading part of self-attention computations offline before inference. Furthermore, we present a unified mapping method according to the computing characteristics of attention-based and convolution-based models. This accelerator implements multi-head attention in Transformer, independent ResNet-50 and hybrid blocks of attention and con-volution in BoTNet-50 at 200MHz on Xilinx Virtex Ultrascale+ XCVU37P. Experimental results show that the solution is nearly 3.62 times more energy-efficient than the NVIDIA V100 GPU, and the computational efficiency is 11.86% and 28.29% higher than the state-of-the-art Transformer and ResNet-50 accelerators, respectively.
10
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia et al. · 2015 · 46.8K citations