ChatGPT outperforms crowd workers for text-annotation tasks

Concepts

ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingAnnotationLanguage ModelingLanguage ModelsAutomated Text ClassificationAutomatic AnnotationFrame DetectionIntercoder Agreement In EvaluationLarge Language ModelsLarge-scale Natural Language ProcessingText AnnotationText Classification

TL;DR

Many NLP tasks rely on manual text annotations, which are typically performed by crowd workers or trained annotators on platforms such as MTurk. In a study of 6,183 tweets and news articles, ChatGPT achieved about 25 percentage points higher zero‑shot accuracy than crowd workers, higher intercoder agreement, and a per‑annotation cost under $0.003—roughly thirty times cheaper—demonstrating that large language models can greatly improve annotation efficiency.

Abstract

Many NLP applications require manual text annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using four samples of tweets and news articles ( n = 6,183), we show that ChatGPT outperforms crowd workers for several annotation tasks, including relevance, stance, topics, and frame detection. Across the four datasets, the zero-shot accuracy of ChatGPT exceeds that of crowd workers by about 25 percentage points on average, while ChatGPT’s intercoder agreement exceeds that of both crowd workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003—about thirty times cheaper than MTurk. These results demonstrate the potential of large language models to drastically increase the efficiency of text classification.