Network Computation in Neural Systems · 1997 · 229 citations · 15 references
ComputingAstronomyForecastingMachine LearningStatistical LearningBias-variance TradeoffEnsemble LearningEnsemble MethodsNeural NetworksEnergy PredictionEnergy Prediction ModelsEnsemble AveragingEnsemble Learning TheoryEnsemble VarianceEnsemble-based PredictionModel AveragingVariance Reduction Techniques
Based on an observation about the different effect of ensemble averaging on the bias and variance portions of the prediction error, we discuss training methodologies for ensembles of networks. We demonstrate the effect of variance reduction and present a method of extrapolation to the limit of an infinite ensemble. A significant reduction of variance is obtained by averaging just over initial conditions of the neural networks, without varying architectures or training sets. The minimum of the ensemble prediction error is reached later than that of a single network. In the vicinity of the minimum, the ensemble prediction error appears to be flatter than that of the single network, thus simplifying optimal stopping decision. The results are demonstrated on sunspots data, where the predictions are among the best obtained, and on the 1993 energy prediction competition data set B.†School of Physics and Astronomy. ury@tarazan.tau.ac.il.‡School of Mathematical Sciences. nin@math.tau.ac.il.§School of Physics and Astronomy. horn@vm.tau.ac.il.
15
Jeffrey L. Elman · Cognitive Science · 1990 · 10.8K citations
David H. Wolpert · Neural Networks · 1992 · 7.5K citations
Spectral Analysis and Time Series
David B. Preston · Technometrics · 1983 · 4.9K citations
Neural Networks and the Bias/Variance Dilemma
Stuart Geman, Elie Bienenstock, René Doursat · Neural Computation · 1992 · 3.6K citations