Proceedings of the AAAI Conference on Artificial Intelligence · 2010 · 125 citations · 11 references
ComputingAutonomous SystemsComputer ScienceArtificial IntelligenceDecision MakingDecision TheoryCollective IntelligenceIntelligent AgentsReinforcement LearningTask SchedulingWorkflow ManagementWorkflow OptimizationCrowdsourcingCrowdsourcing Decision-makingHuman ComputationHuman IntelligenceHuman-ai Decision MakingHuman-in-the-loop Workflow ControlHuman-in-the-loop WorkflowsIterative Improvement
Crowd‑sourcing outsources tasks to unknown workers via open calls, popular on platforms like Amazon Mechanical Turk, and often splits large jobs into iterative subtasks to combine worker outputs into complex workflows. This paper investigates whether an autonomous agent can control such workflows without human intervention, potentially outperforming today’s fixed control programs. The authors introduce TurKontrol, a planner that casts workflow control as a decision‑theoretic optimization problem, balancing solution quality against worker cost and providing a mathematical framework for decision points in common workflow classes. Experimental results show TurKontrol achieves substantially higher utilities than popular fixed‑policy approaches.
Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people ("workers") as an open call (e.g., on Amazon's Mechanical Turk). Crowd-sourcing has become immensely popular with hoards of employers ("requesters"), who use it to solve a wide variety of jobs, such as dictation transcription, content screening, etc. In order to achieve quality results, requesters often subdivide a large task into a chain of bite-sized subtasks that are combined into a complex, iterative workflow in which workers check and improve each other's results. This paper raises an exciting question for AI — could an autonomous agent control these workflows without human intervention, yielding better results than today's state of the art, a fixed control program? We describe a planner, TurKontrol, that formulates workflow control as a decision-theoretic optimization problem, trading off the implicit quality of a solution artifact against the cost for workers to achieve it. We lay the mathematical framework to govern the various decisions at each point in a popular class of workflows. Based on our analysis we implement the workflow control algorithm and present experiments demonstrating that TurKontrol obtains much higher utilities than popular fixed policies.
11
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text
L. von Ahn · Computer · 2006 · 966 citations
A softbot-based interface to the Internet
Oren Etzioni, Daniel S. Weld · Communications of the ACM · 1994 · 455 citations · Full text
Greg Little, Lydia B. Chilton, Max Goldman et al. · 2009 · 239 citations