IEEE Transactions on Circuits and Systems I Regular Papers · 2023 · 17 citations · 27 references
ComputingComputer EngineeringComputer ScienceArtificial IntelligenceComputer ArchitectureMachine LearningAccelerator ArchitectureDeep LearningHardware AccelerationNeural NetworksPattern RecognitionComplexityCordic AlgorithmsDeep Learning AccelerationDeep Learning AcceleratorsHardware ResourcesImplementation ComplexityTraining
Softmax is a common and complex activation function in Deep Neural Networks (DNN). However, it is a challenge to apply it efficiently in DNN training hardware accelerator. Therefore, we propose a high precision calculation method and architecture based on base-2 softmax, which has low hardware complexity than base- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> softmax but can still be useful in DNN training. First, we simplify the hardware implementation complexity of calculating base-2 softmax. Second, we use the base-2 hyperbolic COordinate Rotation Digital Computer (CORDIC) to implement the core computation. Finally, we show that the proposed method can be used in DNN training through experiments. Moreover, with the same order of the magnitude of high precision, our hardware cost is lower than traditional base- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> softmax or other alternative design methods. Under TSMC 28nm CMOS technology, an example design of our architecture has the area of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$98787.43\mu m^{2}$ </tex-math></inline-formula> and the power consumption of 24.72mW for circuit synthesis at the frequency of 1GHz.
27
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
A High-Speed and Low-Complexity Architecture for Softmax Function in Deep Learning
Meiqi Wang, Siyuan Lu, Danyang Zhu et al. · 2018 · 196 citations