Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · 2013 · 64 citations · 13 references
ManagementEconomicsBehavioral EconomicsGame TheoryIncentive SystemsBehavioral IncentivesIncentive AlignmentIncentive SchemesIndividual IncentivesMechanism DesignMonetary IncentivesIncentive Compatible MechanismsIncentive MechanismsIncentive ProvisionOptimal Mechanism DesignPaymentPeer PredictionSelf-selection MechanismsThreshold Setting
Agents are asked to rank two objects in a setting where effort is costly and agents differ in quality (which is the probability that they can identify the correct, ground truth, ranking). We study simple output-agreement mechanisms that pay an agent in the case she agrees with the report of another, and potentially penalizes for disagreement through a negative payment. Assuming access to a quality oracle, able to determine whether an agent's quality is above a given threshold, we design a payment scheme that aligns incentives so that agents whose quality is above this threshold participate and invest effort. Precluding negative payments leads the expected cost of this quality-oracle mechanism to increase by a factor of 2 to 5 relative to allowing both positive and negative payments. Dropping the assumption about access to a quality oracle, we further show that negative payments can be used to make agents with quality lower than the quality threshold choose to not to participate, while those above continue to participate and invest effort. Through the appropriate choice of payments, any design threshold can be achieved. This self-selection mechanism has the same expected cost as the cost-minimal quality-oracle mechanism, and thus when using the self-selection mechanism, perfect screening comes for free.
13
A Bayesian Truth Serum for Subjective Data
Dražen Prelec · Science · 2004 · 545 citations
Tuned Models of Peer Assessment in MOOCs
Chris Piech, Jonathan Huang, Zhenghao Chen et al. · arXiv (Cornell University) · 2013 · 241 citations · Full text
Education, Educational Assessment, Educational Technology +17
A unified framework for dynamic pari-mutuel information market design
Shipra Agrawal, Erick Delage, Mark Peters et al. · 2009 · 167 citations