Can Social Media Listening Platforms’ Artificial Intelligence Be Trusted? Examining the Accuracy of Crimson Hexagon’s (Now Brandwatch Consumer Research’s) AI-Driven Analyses

Abstract

Practitioners and scholars increasingly employ social media listening platforms (SMLPs) driven by artificial intelligence (AI) to extract actionable insights from large amounts of social media data informing research questions and brand strategy. Due to their proprietary nature, AI tools within SMLPs are “black boxes” that force users to accept results on blind faith, a source of concern in industry and academia. This study seeks to provide greater understanding of the strengths and weaknesses of SMLPs by assessing the AI-based results of leading SMLP Crimson Hexagon (now Brandwatch Consumer Research) against those of a standard human content analysis and an analysis conducted using Linguistic Inquiry and Word Count (LIWC). Analyzing a random 10,000-post sample of the conversation around the Nike “Dream Crazy” ad featuring Colin Kaepernick, findings reveal Crimson Hexagon’s AI tools to be woefully unreliable in terms of brand identification as well as detection of post and brand sentiment polarity, specific emotions, and brand outcomes, demonstrating the hazards of blindly relying upon conclusions drawn from black-box social media listening platforms. Findings highlight the need for researchers to examine algorithm documentation and training data sets, as well as assess AI-generated data prior to use in research models and decisions.