Management Science · 2016 · 77 citations · 31 references
ComputingComputer ScienceDecision SciencesDecision AnalysisForecastingMachine LearningEnsemble ForecastingEnsemble LearningEnsemble MethodsForecast EvaluationOverfittingPredictive ModelingCombining ForecastsForecast Performance EvaluationForecasting AccuracyMachine Learning Forecasting ModelsPrediction IntervalsPrediction Intervals CalibrationRandom ForestsRegression Trees
Firms today average forecasts collected from multiple experts and models. Because of cognitive biases, strategic incentives, or the structure of machine-learning algorithms, these forecasts are often overfit to sample data and are overconfident. Little is known about the challenges associated with aggregating such forecasts. We introduce a theoretical model to examine the combined effect of overfitting and overconfidence on the average forecast. Their combined effect is that the mean and median probability forecasts are poorly calibrated with hit rates of their prediction intervals too high and too low, respectively. Consequently, we prescribe the use of a trimmed average, or trimmed opinion pool, to achieve better calibration. We identify the random forest, a leading machine-learning algorithm that pools hundreds of overfit and overconfident regression trees, as an ideal environment for trimming probabilities. Using several known data sets, we demonstrate that trimmed ensembles can significantly improve the random forest’s predictive accuracy. This paper was accepted by James Smith, decision analysis.
31
Leo Breiman · Machine Learning · 2001 · 126.2K citations · Full text
Multivariate Adaptive Regression Splines
Jerome H. Friedman · The Annals of Statistics · 1991 · 8.1K citations · Full text
Big Data: New Tricks for Econometrics
Hal R. Varian · The Journal of Economic Perspectives · 2014 · 1.5K citations · Full text
Comparison Methods for Stochastic Models and Risks
Mark Anthony McComb · Technometrics · 2003 · 1.3K citations