Efficient hardware architecture of softmax layer in deep neural network

Deep neural network (DNN) has emerged as a very important machine learning and pattern recognition technique in the big data era. Targeting to different types of training and inference tasks, the structure of DNN varies with flexible choices of different component layers, such as fully connection layer, convolutional layer, pooling layer and softmax layer. Deviated from other layers that only require simple operations like addition or multiplication, the softmax layer contains expensive exponentiation and division, thereby causing the hardware design of softmax layer suffering from high complexity, long critical path delay and overflow problems. This paper, for the first time, presents efficient hardware architecture of softmax layer in DNN. By utilizing the domain transformation technique and down-scaling approach, the proposed hardware architecture avoids the aforementioned problems. Analysis shows that the proposed hardware architecture achieves reduced hardware complexity and critical path delay.