Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise

TLDR

Computer vision systems increasingly rely on massive hand‑labeled image databases, yet crowdsourced labeling introduces challenges such as unknown and potentially adversarial expertise, varying image difficulty, and the need to combine multiple labels, motivating principled probabilistic solutions. The authors aim to develop a probabilistic model that simultaneously infers image labels, labeler expertise, and image difficulty. The model jointly estimates these quantities by integrating multiple noisy labels within a probabilistic framework. Experiments on simulated and real data show the model outperforms majority vote and remains robust to noisy and adversarial labelers.

Abstract

Modern machine learning-based approaches to computer vision require very large databases of hand labeled images. Some contemporary vision systems already require on the order of millions of images for training (e.g., Omron face detector [9]). New Internet-based services allow for a large number of labelers to collaborate around the world at very low cost. However, using these services brings interesting theoretical and practical challenges: (1) The labelers may have wide ranging levels of expertise which are unknown a priori, and in some cases may be adversarial; (2) images may vary in their level of difficulty; and (3) multiple labels for the same image must be combined to provide an estimate of the actual label of the image. Probabilistic approaches provide a principled way to approach these problems. In this paper we present a probabilistic model and use it to simultaneously infer the label of each image, the expertise of each labeler, and the difficulty of each image. On both simulated and real data, we demonstrate that the model outperforms the commonly used Majority Vote heuristic for inferring image labels, and is robust to both noisy and adversarial labelers.