Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose

TLDR

Twitter’s Streaming API offers a sampled view of the platform’s billions of tweets, but its undocumented sampling process raises doubts about whether it accurately reflects overall activity. This study aims to determine whether data from the Streaming API is a valid representation of Twitter’s full activity by comparing it to the Firehose. We collected parallel datasets from the Streaming API and the Firehose and compared them using statistical, topical, network, and geographic metrics. Our analysis shows that the Streaming API’s sample differs significantly from the Firehose, highlighting limitations for researchers relying on it.

Abstract

Twitter is a social media giant famous for the exchange of short, 140-character messages called "tweets". In the scientific community, the microblogging site is known for openness in sharing its data. It provides a glance into its millions of users and billions of tweets through a "Streaming API" which provides a sample of all tweets matching some parameters preset by the API user. The API service has been used by many researchers, companies, and governmental institutions that want to extract knowledge in accordance with a diverse array of questions pertaining to social media. The essential drawback of the Twitter API is the lack of documentation concerning what and how much data users get. This leads researchers to question whether the sampled data is a valid representation of the overall activity on Twitter. In this work we embark on answering this question by comparing data collected using Twitter's sampled API service with data collected using the full, albeit costly, Firehose stream that includes every single published tweet. We compare both datasets using common statistical metrics as well as metrics that allow us to compare topics, networks, and locations of tweets. The results of our work will help researchers and practitioners understand the implications of using the Streaming API.