Internet-scale collection of human-reviewed data

TLDR

Enterprise and web data processing systems rely on human‑reviewed data for training and monitoring machine learning models, yet current practices depend on in‑house or offshore efforts, prompting emerging solutions that aim to automate large‑scale collection. The study aims to evaluate the effectiveness of an automated human‑reviewed data collection application. The authors performed extensive experiments and assessed the feasibility of leveraging Yahoo!

Abstract

Enterprise and web data processing and content aggregation systems often require extensive use of human-reviewed data (e.g. for training and monitoring machine learning-based applications). Today these needs are often met by in-house efforts or out-sourced offshore contracting. Emerging applications attempt to provide automated collection of human-reviewed data at Internet-scale. We conduct extensive experiments to study the effectiveness of one such application. We also study the feasibility of using Yahoo! Answers, a general question-answering forum, for human-reviewed data collection.