medRxiv · 2023 · 16 citations · 15 references
MedicineMedical EducationOtolaryngologyClinical InformaticsClinical MedicineDiagnostic AssessmentClinical Natural Language ProcessingMedical Technology AssessmentClinical Question AnsweringConcordance AssessmentLarge Language ModelsModel Performance AnalysisModel Performance EvaluationMultiple-choice ExaminationsStandardized ExamsWritten Examinations
Abstract Background ChatGPT is among the most popular Large Language Models (LLM), exhibiting proficiency in various standardized tests, including multiple-choice medical board examinations. However, its performance on Otolaryngology–Head and Neck Surgery (OHNS) board exams and open-ended medical board examinations has not been reported. We present the first evaluation of LLM (ChatGPT-4) on such examinations and propose a novel method to assess an artificial intelligence (AI) model’s performance on open-ended medical board examination questions. Methods Twenty-one open end questions were adopted from the Royal College of Physicians and Surgeons of Canada’s sample exam to query ChatGPT-4 on April 11th, 2023, with and without prompts. A new CVSA (concordance, validity, safety, and accuracy) model was developed to evaluate its performance. Results In an open-ended question assessment, ChatGPT-4 achieved a passing mark (an average of 75% across three trials) in the attempts. The model demonstrated high concordance (92.06%) and satisfactory validity. While demonstrating considerable consistency in regenerating answers, it often provided only partially correct responses. Notably, concerning features such as hallucinations and self-conflicting answers were observed. Conclusion ChatGPT-4 achieved a passing score in the sample exam, and demonstrated the potential to pass the Canadian Otolaryngology–Head and Neck Surgery Royal College board examination. Some concerns remain due to its hallucinations that could pose risks to patient safety. Further adjustments are necessary to yield safer and more accurate answers for clinical implementation.
15
Evaluating the Performance of ChatGPT in Ophthalmology
Fares Antaki, Samir Touma, Daniel Milad et al. · Ophthalmology Science · 2023 · 494 citations · Full text