2017 · 28 citations · 10 references
ComputingComputer ScienceArtificial IntelligenceComputer VisionMachine LearningDeep LearningImage ClassificationNeural NetworksPattern RecognitionActivationActivation FunctionsConvolutional Neural NetworksGradient DescentVanishing Gradient Problem
This article proposes “Hexpo”, an activation function that has ability to scale the gradient and hence overcome the vanishing gradient problem. Unlike rectified linear units family which produces identity mapping on positive inputs, Hexpo has scalable limits on both positive and negative values. With parametrization, the active domain of Hexpo and the output it maps to is flexible. Thus it can alleviate the vanishing gradient problem from both gradient flow and local gradient aspect, while preserving the upper and lower bound of output. Parametrization also offers Hexpo the ability to produce the output that is close to zero. On experiments involving MNIST hand digit recognition dataset, Hexpo outperforms rectified linear units family(rectified linear unit and exponential linear unit) by the accuracy and speed of learning. On the experiments involving CIFAR-10 tiny image recognition dataset and convolutional layers, Hexpo outperforms rectified linear unit and performs similar in comparison with exponential linear unit.
10
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Learning representations by back-propagating errors
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · Nature · 1986 · 30.9K citations
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2015 · 18.8K citations
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh · Neural Computation · 2006 · 16.4K citations