Transactions of the Japanese Society for Artificial Intelligence · 2013 · 27 citations · 14 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningNatural Language ProcessingLatent Variable ModelsSupervised LearningCrowdsourced Data CollectionCrowdsourcing
Crowdsourcing services are often used to collect a large amount of labeled data for machine learning. Although they provide us an easy way to get labels at very low cost in a short period, they have serious limitations. One of them is the variable quality of the crowd-generated data. There have been many attempts to increase the reliability of crowd-generated data and the quality of classifiers obtained from such data. However, in these problem settings, relatively few researchers have tried using expert-generated data to achieve further improvements. In this paper, we apply three models that deal with the problem of learning from crowds to this problem: a latent class model, a personal classifier model, and a data-dependent error model. We evaluate these methods against two baseline methods on a real data set to demonstrate the effectiveness of combining crowd-generated data and expert-generated data.
14
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text
Named Entity Recognition in Tweets: An Experimental Study
Alan Ritter, Sam Clark, Oren Etzioni · 2011 · 1.2K citations
Vikas C. Raykar, Shipeng Yu, Linda Zhao et al. · Journal of Machine Learning Research · 2010 · 1K citations