Using prediction markets to estimate the reproducibility of scientific research

TLDR

Increasing concern about reproducibility of scientific research is compounded by the lack of rapid mechanisms to flag findings unlikely to replicate. The study proposes using prediction markets to bridge the gap in identifying findings unlikely to replicate. The authors employ prediction markets to estimate the reproducibility of studies. Irreproducible preclinical research costs US$28 billion annually, and prediction markets accurately forecast replication outcomes for 44 psychology studies, outperforming individual surveys.

Abstract

Significance There is increasing concern about the reproducibility of scientific research. For example, the costs associated with irreproducible preclinical research alone have recently been estimated at US$28 billion a year in the United States. However, there are currently no mechanisms in place to quickly identify findings that are unlikely to replicate. We show that prediction markets are well suited to bridge this gap. Prediction markets set up to estimate the reproducibility of 44 studies published in prominent psychology journals and replicated in The Reproducibility Project: Psychology predict the outcomes of the replications well and outperform a survey of individual forecasts.