Frontiers in Digital Health · 2025 · 22 citations · 23 references
ComputingComputer ScienceHealth InformaticsArtificial IntelligenceClinical Decision SupportPain ManagementClinical Natural Language ProcessingPain AssessmentChatbot InteractionChatbot SystemsChatbotsClinical Guidelines AdherenceClinical Practice Guidelines ImplementationIntelligent Virtual AssistantsLumbosacral DiseaseRadicular Pain
Introduction: Artificial Intelligence (AI) chatbots, which generate human-like responses based on extensive data, are becoming important tools in healthcare by providing information on health conditions, treatments, and preventive measures, acting as virtual assistants. However, their performance in aligning with clinical practice guidelines (CPGs) for providing answers to complex clinical questions on lumbosacral radicular pain is still unclear. We aim to evaluate AI chatbots' performance against CPG recommendations for diagnosing and treating lumbosacral radicular pain. Methods: We performed a cross-sectional study to assess AI chatbots' responses against CPGs recommendations for diagnosing and treating lumbosacral radicular pain. Clinical questions based on these CPGs were posed to the latest versions (updated in 2024) of six AI chatbots: ChatGPT-3.5, ChatGPT-4o, Microsoft Copilot, Google Gemini, Claude, and Perplexity. The chatbots' responses were evaluated for (a) consistency of text responses using Plagiarism Checker X, (b) intra- and inter-rater reliability using Fleiss' Kappa, and (c) match rate with CPGs. Statistical analyses were performed with STATA/MP 16.1. Results: We found high variability in the text consistency of AI chatbot responses (median range 26%-68%). Intra-rater reliability ranged from "almost perfect" to "substantial," while inter-rater reliability varied from "almost perfect" to "moderate." Perplexity had the highest match rate at 67%, followed by Google Gemini at 63%, and Microsoft Copilot at 44%. ChatGPT-3.5, ChatGPT-4o, and Claude showed the lowest performance, each with a 33% match rate. Conclusions: Despite the variability in internal consistency and good intra- and inter-rater reliability, the AI Chatbots' recommendations often did not align with CPGs recommendations for diagnosing and treating lumbosacral radicular pain. Clinicians and patients should exercise caution when relying on these AI models, since one to two-thirds of the recommendations provided may be inappropriate or misleading according to specific chatbots.
23
The Measurement of Observer Agreement for Categorical Data
J. Richard Landis, Gary G. Koch · Biometrics · 1977 · 78.9K citations · Full text
Erik von Elm, Douglas G. Altman, Matthias Egger et al. · Journal of Clinical Epidemiology · 2008 · 16.4K citations · Full text
Public Health, Public Health Practice, Epidemiologic Methods +10
Lidwine B. Mokkink, Caroline B. Terwee, Donald L. Patrick et al. · Journal of Clinical Epidemiology · 2010 · 4.4K citations · Full text
Biostatistics: The Bare Essentials
Naomi Vaisrub · JAMA · 2009 · 675 citations