Algorithmic content moderation: Technical and political challenges in the automation of platform governance

TLDR

Government pressure on major technology companies has spurred firms and legislators to seek technical solutions for platform governance challenges such as hate speech and misinformation, leading to the widespread deployment of algorithmic moderation systems at scale. This article offers an accessible technical primer on algorithmic moderation, reviews the tools used by major platforms to address copyright infringement, terrorism, and toxic speech, and highlights the key political and ethical issues that arise as reliance on these systems grows. It examines the technical components of algorithmic moderation—automated hash‑matching and predictive machine learning models—employed by platforms like Facebook, YouTube, and Twitter to process user‑generated content. The authors find that while algorithmic moderation is increasingly necessary to meet public expectations for platform responsibility, safety, and security, these systems remain opaque, unaccountable, and poorly understood, and even well‑optimized algorithms can worsen opacity, fairness, and the political nature of speech decisions.

Abstract

As government pressure on major technology companies builds, both firms and legislators are searching for technical solutions to difficult platform governance puzzles such as hate speech and misinformation. Automated hash-matching and predictive machine learning tools – what we define here as algorithmic moderation systems – are increasingly being deployed to conduct content moderation at scale by major platforms for user-generated content such as Facebook, YouTube and Twitter. This article provides an accessible technical primer on how algorithmic moderation works; examines some of the existing automated tools used by major platforms to handle copyright infringement, terrorism and toxic speech; and identifies key political and ethical issues for these systems as the reliance on them grows. Recent events suggest that algorithmic moderation has become necessary to manage growing public expectations for increased platform responsibility, safety and security on the global stage; however, as we demonstrate, these systems remain opaque, unaccountable and poorly understood. Despite the potential promise of algorithms or ‘AI’, we show that even ‘well optimized’ moderation systems could exacerbate, rather than relieve, many existing problems with content policy as enacted by platforms for three main reasons: automated moderation threatens to (a) further increase opacity, making a famously non-transparent set of practices even more difficult to understand or audit, (b) further complicate outstand- ing issues of fairness and justice in large-scale sociotechnical systems and (c) re-obscure the fundamentally political nature of speech decisions being executed at scale.