International Journal of Medical Informatics · 2025 · 96 citations · 20 references
ComputingComputer ScienceSurgeryArtificial IntelligenceClinical InformaticsMachine LearningGenerative Ai In EducationMedical Technology AssessmentNeural NetworksSurgery EducationAi-generated Learning MaterialsAutomated Performance MetricsComparative Data AnalysisPatient Education Materials EvaluationSurgical Simulation TrainingSurgical Skill AssessmentSurgical Training
BACKGROUND: Access to patient-centered health information is essential for informed decision-making. However, online medical resources vary in quality and often fail to accommodate differing degrees of health literacy. This issue is particularly evident in surgical contexts, where complex terminology obstructs patient comprehension. With the increasing reliance on AI models for supplementary medical information, the reliability and readability of AI-generated content require thorough evaluation. OBJECTIVE: This study aimed to evaluate four natural language processing models-ChatGPT-4o, ChatGPT-o3 mini, DeepSeek-V3, and DeepSeek-R1-in generating patient education materials for three common spinal surgeries: lumbar discectomy, spinal fusion, and decompressive laminectomy. Information quality was evaluated using the DISCERN score, and readability was assessed through Flesch-Kincaid indices. RESULTS: DeepSeek-R1 produced the most readable responses, with Flesch-Kincaid Grade Level (FKGL) scores ranging from 7.2 to 9.0, succeeded by ChatGPT-4o. In contrast, ChatGPT-o3 exhibited the lowest readability (FKGL > 10.4). The DISCERN scores for all AI models were below 60, classifying the information quality as "fair," primarily due to insufficient cited references. CONCLUSION: All models achieved merely a "fair" quality rating, underscoring the necessity for improvements in citation practices, and personalization. Nonetheless, DeepSeek-R1 and ChatGPT-4o generated more readable surgical information than ChatGPT-o3. Given that enhanced readability can improve patient engagement, reduce anxiety, and contribute to better surgical outcomes, these two models should be prioritized for assisting patients in the clinical. LIMITATION & FUTURE DIRECTION: This study is limited by the rapid evolution of AI models, its exclusive focus on spinal surgery education, and the absence of real-world patient feedback, which may affect the generalizability and long-term applicability of the findings. Future research ought to explore interactive, multimodal approaches and incorporate patient feedback to ensure that AI-generated health information is accurate, accessible, and facilitates informed healthcare decisions.
20
Rudolph Flesch · Journal of Applied Psychology · 1948 · 5.2K citations
David Charnock, Sasha Shepperd, Gill Needham et al. · Journal of Epidemiology & Community Health · 1999 · 3.1K citations · Full text
Social Sciences, Health Information Management, Evidence-based Practice +15