CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities

Mina Lee, Percy Liang, Qian Yang

CHI Conference on Human Factors in Computing Systems · 2022 · 321 citations · 29 references

DOIFull text

Open access

Concepts

TL;DR

Large language models provide unprecedented language generation capabilities, yet their highly context‑dependent performance is difficult to quantify and often subjectively interpreted. The paper argues that curating and analyzing large interaction datasets enables the HCI community to conduct more incisive examinations of LMs’ generative capabilities. CoAuthor is a dataset of 1,445 writing sessions involving 63 writers and four GPT‑3 instances, capturing rich interactions to reveal GPT‑3’s capabilities in creative and argumentative writing. The dataset demonstrates GPT‑3’s language, ideation, and collaboration abilities, shows its role as a writing collaborator under various definitions, and is publicly available with a replay interface at https://coauthor.stanford.edu.

Abstract

Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs’ generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3’s capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3’s language, ideation, and collaboration capabilities, and reveal its contribution as a writing “collaborator” under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs’ promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.

References

29