Journal of New Results in Science · 2021 · 61 citations · 28 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningNatural Language ProcessingDeep LearningMachine Learning For Big DataNeural NetworksPattern RecognitionRepresentation LearningAccuracy-computational Cost TradeoffActivationActivation FunctionsDeep Learning For Big DataDeep Learning For ForecastingDeep Learning-based ForecastingFinancial Time Series Forecasting
In deep learning models, the inputs to the network are processed using activation functions to generate the output corresponding to these inputs. Deep learning models are of particular importance in analyzing big data with numerous parameters and forecasting and are useful for image processing, natural language processing, object recognition, and financial forecasting. Also, in deep learning algorithms, activation functions have been developed by taking into account features such as performing the learning process in a healthy way, preventing excessive learning, increasing the accuracy performance, and reducing the computational cost. In this study, we present an overview of common and current activation functions used in deep learning algorithms. In the study, fixed and trainable activation functions are introduced. As fixed activation functions, sigmoid, hyperbolic tangent, ReLU, softplus and swish, and as trainable activation functions, LReLU, ELU, SELU and RSigELU are introduced.
28
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2015 · 18.8K citations