A Better Activation Function for Artificial Neural Networks

Abstract

An activation function, possibly new, is proposed for use in digital simulation of arti#cial neural networks, on the ground that the computational operation count for this function is much smaller than for those employing exponentials and it satis#es a simple di#erential equation generalizing the logistic equation. Introduction: activation functions In the digital simulation of neural networks a feedforward memoryless neuron is represented by the input-output relation y = ## P n 1 w k x k # where ####, the sigmoidal activation or #squash" function, should have the property that it is positive monotone between the values ,1 and 1 #or between 0 and 1# for u 2 #,1;1#. For use in #nding optimal weights w k to minimize jjy , y desired jj 2 by backpropagation #gradient# algorithms, it is also a requirement that #### be di#erentiable and that it satisfy a simple di#erential equation, thus permitting the evaluation of increments of weights via the chain rule for partial derivatives. In t...