Proceedings of the AAAI Conference on Artificial Intelligence · 2013 · 117 citations · 28 references
ComputingComputer ScienceSocial PsychologyAlgorithmsArtificial IntelligenceDecision TheoryElectionsMajority RuleSocial ComputingVoting BehaviorVoting RulesVoting TheoryComputational Social ChoiceCrowdsourcingCrowdsourcing Decision-makingHuman ComputationHuman-ai Decision SupportHuman-algorithm InteractionSocial ComputationVoting Decision Making
Designers of human computation systms often face the need to aggregate noisy information provided by multiple people. While voting is often used for this purpose, the choice of voting method is typically not principled. We conduct extensive experiments on Amazon Mechanical Turk to better understand how different voting rules perform in practice. Our empirical conclusions show that noisy human voting can differ from what popular theoretical models would predict. Our short-term goal is to motivate the design of better human computation systems; our long-term goal is to spark an interaction between researchers in (computational) social choice and human computation.
28
John G. Cragg, Paul Zarembka · Canadian Journal of Economics/Revue canadienne d économique · 1975 · 8.4K citations
A law of comparative judgment.
L. L. Thurstone · Psychological Review · 1927 · 5.2K citations
Rank aggregation methods for the Web
Cynthia Dwork, Ravi Kumar, Moni Naor et al. · 2001 · 1.8K citations
Predicting protein structures with a multiplayer online game
Seth Cooper, Firas Khatib, Adrien Treuille et al. · Nature · 2010 · 1.4K citations