Incentive mechanism and protocol design for crowdsourcing systems

Abstract

Crowdsourcing systems such as Amazon Mechanical Turk, Yahoo!Answers, and Google Helpouts have attracted extensive attention over the past few years. In a crowdsourcing system, a large group of “workers” solve the tasks outsourced by “requesters”. To make a crowdsourcing system sustainable, it is vital to attract users (both requesters and workers) to participate, and incentivize high-quality solutions. To achieve this objective, we design an effective incentive mechanism and reputation protocol. Our design incorporates various important elements of a crowdsourcing system such as workers having heterogeneous skill sets (i.e., some are “experts” while others are “novices”), and task assignment process, rating system, etc. Our incentive mechanism is composed of a rating system and a reward dividing scheme, and requires the system administrator to divide the reward based on requesters' rating on the solution quality. We derive the minimum reward needed so that “expert” workers are guaranteed provide high-quality solutions. We show that “novice” workers provide low-quality solutions, and our reputation protocol eliminates this undesirable behavior by tracking a worker's solution history and penalizing him when his reputation is poor. We apply repeated game-theoretic frameworks to quantify the impact of this reputation protocol on requesters' cost in guaranteeing high quality solutions.