Racial disparities in automated speech recognition

TLDR

Automated speech recognition systems, now ubiquitous in virtual assistants, closed captioning, and healthcare dictation, have improved with deep learning and large datasets, yet there is concern they may not perform equally across population subgroups. This study evaluates the transcription accuracy of five leading ASR systems on structured interviews from 42 white and 73 black speakers. The evaluation uses a 19.8‑hour corpus of matched‑age and gender interviews from five U.S. cities, testing Amazon, Apple, Google, IBM, and Microsoft ASR engines.

Abstract

Automated speech recognition (ASR) systems, which use sophisticated machine-learning algorithms to convert spoken language to text, have become increasingly widespread, powering popular virtual assistants, facilitating automated closed captioning, and enabling digital dictation platforms for health care. Over the last several years, the quality of these systems has dramatically improved, due both to advances in deep learning and to the collection of large-scale datasets used to train the systems. There is concern, however, that these tools do not work equally well for all subgroups of the population. Here, we examine the ability of five state-of-the-art ASR systems—developed by Amazon, Apple, Google, IBM, and Microsoft—to transcribe structured interviews conducted with 42 white speakers and 73 black speakers. In total, this corpus spans five US cities and consists of 19.8 h of audio matched on the age and gender of the speaker. We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers. We trace these disparities to the underlying acoustic models used by the ASR systems as the race gap was equally large on a subset of identical phrases spoken by black and white individuals in our corpus. We conclude by proposing strategies—such as using more diverse training datasets that include African American Vernacular English—to reduce these performance differences and ensure speech recognition technology is inclusive.