Does the “surprisingly popular” method yield accurate crowdsourced predictions?

Abstract

The "surprisingly popular" method (SP) of aggregating individual judgments has shown promise in overcoming a weakness of other crowdsourcing methods-situations in which the majority is incorrect. This method relies on participants' estimates of other participants' judgments; when an option is chosen more often than the average metacognitive judgments of that option, it is "surprisingly popular" and is selected by the method. Although SP has been shown to improve group decision making about factual propositions (e.g., state capitals), its application to future outcomes has been limited. In three preregistered studies, we compared SP to other methods of aggregating individual predictions about future events. Study 1 examined predictions of football games, Study 2 examined predictions of the 2018 US midterm elections, and Study 3 examined predictions of basketball games. When applied to judgments made by objectively assessed experts, SP performed slightly better than other aggregation methods. Although there is still more to learn about the conditions under which SP is effective, it shows promise as a means of crowdsourcing predictions of future outcomes.