A Timely Intervention: Tracking the Changing Meanings of Political Concepts with Word Vectors

TLDR

Word vectorization offers a powerful way to automate semantic analysis, yet its success has been limited to large corpora, while political science studies often involve small datasets where word meanings shift over time, posing methodological challenges. The study asks whether word vectors can track the evolving cultural meanings of words in typical small‑corpus settings. Four time‑sensitive word2vec implementations were evaluated against a gold standard derived from 161 years of newspaper coverage. One implementation consistently matched human judgments of how public discourse on equality has changed, and the author recommends best practices such as bootstrap resampling and pretraining, demonstrating that word2vec enables finer‑grained analysis of meaning change than other text‑as‑data methods.

Abstract

Word vectorization is an emerging text-as-data method that shows great promise for automating the analysis of semantics—here, the cultural meanings of words—in large volumes of text. Yet successes with this method have largely been confined to massive corpora where the meanings of words are presumed to be fixed. In political science applications, however, many corpora are comparatively small and many interesting questions hinge on the recognition that meaning changes over time. Together, these two facts raise vexing methodological challenges. Can word vectors trace the changing cultural meanings of words in typical small corpora use cases? I test four time-sensitive implementations of word vectors ( word2vec ) against a gold standard developed from a modest data set of 161 years of newspaper coverage. I find that one implementation method clearly outperforms the others in matching human assessments of how public dialogues around equality in America have changed over time. In addition, I suggest best practices for using word2vec to study small corpora for time series questions, including bootstrap resampling of documents and pretraining of vectors. I close by showing that word2vec allows granular analysis of the changing meaning of words, an advance over other common text-as-data methods for semantic research questions.