1996 · 146 citations · 7 references
ComputingComputer ScienceMachine LearningOptimizationStatistical LearningDigital Signal ProcessingMetric LearningNeural NetworksOptimization For Machine LearningRiemannian GeometrySignal ProcessingIndependent Component AnalysisMetric StructureMultilayer PerceptronOptimization AlgorithmsPerceptronRiemannian MetricSteepest DescentStochastic Gradient Descent
The parameter space of neural networks has the Riemannian metric structure. The natural Riemannian gradient should be used instead of the conventional gradient, since the former denotes the steepest descent direction of a loss function in the Riemannian space. The behavior of the stochastic gradient learning algorithm is much more effective if the natural gradient is used. The present paper studies the information-geometrical structure of perceptrons and other networks, and prove that the on-line learning method based on the natural gradient is asymptotically as efficient as the optimal batch algorithm. Adaptive modification of the learning constant is proposed and analyzed in terms of the Riemannian measure and is shown to be efficient. The natural gradient is finally applied to blind separation of mixtured independent signal sources. 1 Introduction Neural learning takes place in the parameter space of modifiable synaptic weights of a neural network. The role of each parameter is dif...
7
A New Learning Algorithm for Blind Signal Separation
Шун-ичи Амари, Andrzej Cichocki, Howard H. Yang · 1995 · 1.9K citations · Full text
Equivariant adaptive source separation
J.-F. Cardoso, B.H. Laheld · IEEE Transactions on Signal Processing · 1996 · 1.4K citations
A Theory of Adaptive Pattern Classifiers
IEEE Transactions on Electronic Computers · 1967 · 529 citations
Information geometry of the EM and em algorithms for neural networks
Neural Networks · 1995 · 341 citations · Full text
Learning Internal Representations
1995 · 205 citations