Communications of the ACM · 2015 · 114 citations · 16 references
ComputingComputer ScienceComputational LinguisticsHuman-computer InteractionNatural Language ProcessingCollaborative WritingComputer-supported Cooperative WorkConversational User InterfacesTextual EditingAuthoring Tools DesignCollaborative Document EditingCrowdsourcingWork - Time Tradeoff
Writing and editing involve complex conceptual and pragmatic tasks, and while authoring tools assist with pragmatics, writers often seek higher‑level help from others. The paper presents Soylent, a word‑processing interface that lets writers invoke Mechanical Turk workers to shorten, proofread, and edit documents on demand. Soylent integrates crowdsourced contributions into the UI using architectural and interaction patterns, and employs a Find‑Fix‑Verify crowd‑programming approach that splits tasks into generation and review stages to improve worker quality. Evaluation studies demonstrate that crowdsourced editing is feasible and analyze its reliability, cost, wait time, and work time.
This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.
16
Hiroshi Ishii, Brygg Ullmer · 1997 · 3.8K citations
Arts, Computer Science, Media Arts +17
Awareness and coordination in shared workspaces
Paul Dourish, Victoria Bellotti · 1992 · 2.5K citations · Full text
Labeling images with a computer game
Luis von Ahn, Laura Dabbish · 2004 · 2.2K citations
Crowdsourcing user studies with Mechanical Turk
Aniket Kittur, Ed H., Bongwon Suh · 2008 · 2K citations
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text