IEEE Transactions on Neural Networks and Learning Systems · 2019 · 39 citations · 42 references
ComputingComputer ScienceData MiningFeature EngineeringMachine LearningAnnotationLabeling SystemsAggregationConceptual LabelingCrowdsourcingGuidanceLabel Noise ModelingLabels
With online crowdsourcing platforms, labels can be acquired at relatively low costs from massive nonexpert workers. To improve the quality of labels obtained from these imperfect crowdsourced workers, we usually let different workers provide labels for the same instance. Then, the true labels for all instances are estimated from these multiple noisy labels. This traditional general-purpose label aggregation process, solely relying on the collected noisy labels, cannot significantly improve the accuracy of integrated labels under a low labeling quality circumstance. This paper proposes a novel bilayer collaborative clustering (BLCC) method for the label aggregation in crowdsourcing. BLCC first generates the conceptual-level features for the instances from their multiple noisy labels and infers the initially integrated labels by performing clustering on the conceptual-level features. Then, it performs another clustering on the physical-level features to form the estimations of the true labels on the physical layer. The clustering results on both layers can facilitate in tracking the changes in the uncertainties of the instances. Finally, the initially integrated labels that are likely to be wrongly inferred on the conceptual layer can be addressed using the estimated labels on the physical layer. The clustering processes on both layers can keep providing guidance information for each other in the multiple label remedy rounds. The experimental results on 12 real-world crowdsourcing data sets show that the performance of the proposed method in terms of accuracy is better than that of the state-of-the-art methods.
42
Vikas C. Raykar, Shipeng Yu, Linda Zhao et al. · Journal of Machine Learning Research · 2010 · 1K citations
The Multidimensional Wisdom of Crowds
Peter Welinder, Steve Branson, Pietro Perona et al. · CaltechAUTHORS (California Institute of Technology) · 2010 · 739 citations · Full text
Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation
Qi Li, Yaliang Li, Jing Gao et al. · 2014 · 491 citations