Management Science · 2021 · 26 citations · 26 references
ComputingComputer ScienceDecision SciencesArtificial IntelligenceForecastingMachine LearningBias-variance TradeoffForecast EvaluationForecasting MethodsMeta-learningPredictive ModelingProbabilistic ForecastingDecision ProblemsExpert Confidence AssessmentForecast Bias DetectionMachine Learning Forecasting ModelsWeighting Of Expert Opinions
Modern forecasting algorithms use the wisdom of crowds to produce forecasts better than those of the best identifiable expert. However, these algorithms may be inaccurate when crowds are systematically biased or when expertise varies substantially across forecasters. Recent work has shown that meta-predictions—a forecast of the average forecasts of others—can be used to correct for biases even when no external information, such as forecasters’ past performance, is available. We explore whether meta-predictions can also be used to improve forecasts by identifying and leveraging the expertise of forecasters. We develop a confidence-based version of the Surprisingly Popular algorithm proposed by Prelec, Seung, and McCoy. As with the original algorithm, our new algorithm is robust to bias. However, unlike the original algorithm, our version is predicted to always weight forecasters with more informative private signals more than forecasters with less informative ones. In a series of experiments, we find that the modified algorithm does a better job in weighting informed forecasters than the original algorithm and show that individuals who are correct more often on similar decision problems contribute more to the final decision than other forecasters. Empirically, the modified algorithm outperforms the original algorithm for a set of 500 decision problems. This paper was accepted by Yan Chen, decision analysis.
26
Combining forecasts: A review and annotated bibliography
Robert T. Clemen · International Journal of Forecasting · 1989 · 2.2K citations
Business, Economics, Meteorology +10
Equivalent Comparisons of Experiments
David Blackwell · The Annals of Mathematical Statistics · 1953 · 1.5K citations · Full text
Experts in uncertainty: opinion and subjective probability in science
Choice Reviews Online · 1992 · 1.2K citations
Business, Psychology, Epistemology +11
Francis Galton · Nature · 1907 · 1.1K citations · Full text