Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis

TLDR

News organizations are widely believed to exhibit ideological bias, yet quantifying such slant has proven methodologically challenging. The study investigates how fifteen major U.S. news outlets select and frame political issues. Using supervised learning to identify political articles from 803,146 stories and crowdsourcing 749 judges to classify 10,502 of them by topic and ideology, the authors combine machine‑learning and human judgment.

Abstract

It is widely thought that news organizations exhibit ideological bias, but rigorously quantifying such slant has proven methodologically challenging. Through a combination of machine-learning and crowdsourcing techniques, we investigate the selection and framing of political issues in fifteen major US news outlets. Starting with 803,146 news stories published over twelve months, we first used supervised learning algorithms to identify the 14 percent of articles pertaining to political events. We then recruited 749 online human judges to classify a random subset of 10,502 of these political articles according to topic and ideological position. Our analysis yields an ideological ordering of outlets consistent with prior work. However, news outlets are considerably more similar than generally believed. Specifically, with the exception of political scandals, major news organizations present topics in a largely nonpartisan manner, casting neither Democrats nor Republicans in a particularly favorable or unfavorable light. Moreover, again with the exception of political scandals, little evidence exists of systematic differences in story selection, with all major news outlets covering a wide variety of topics with frequency largely unrelated to the outlet's ideological position. Finally, news organizations express their ideological bias not by directly advocating for a preferred political party, but rather by disproportionately criticizing one side, a convention that further moderates overall differences.