Incentives to Counter Bias in Human Computation

Abstract

In online labor platforms such as Amazon Mechanical Turk, a good strategy to obtain quality answers is to take aggregate answers submitted by multiple workers, exploiting the wisdom of crowds. However, human computation issusceptible to systematic biases which cannot be corrected by using multiple workers.We investigate a game-theoretic bonus scheme, called peer truth serum (PTS), to overcome this problem. We report on the design and outcomes of a set of experiments to validate this scheme. Results show peer truth serum can indeed correct the biases and increase the answer accuracy by up to 80%.