Arthroscopy The Journal of Arthroscopic and Related Surgery · 2024 · 14 citations · 15 references
ComputingOrthopaedic SurgeryOrthopedic SurgeryClinical InformaticsNatural Language ProcessingOrthopedicsClinical Natural Language ProcessingOrthopedic ExaminationQuestion AnsweringClinical Question AnsweringClinical Question Answering SystemsClinical Reasoning SupportComparative Performance EvaluationQuestion Answering In HealthQuestion Answering Systems Evaluation
PURPOSE: To assess ChatGPT's, Bard's, and Bing Chat's ability to generate accurate orthopaedic diagnoses or corresponding treatments by comparing their performance on the Orthopaedic In-Training Examination (OITE) with that of orthopaedic trainees. METHODS: OITE question sets from 2021 and 2022 were compiled to form a large set of 420 questions. ChatGPT (GPT-3.5), Bard, and Bing Chat were instructed to select one of the provided responses to each question. The accuracy of composite questions was recorded and comparatively analyzed to human cohorts including medical students and orthopaedic residents, stratified by postgraduate year (PGY). RESULTS: ChatGPT correctly answered 46.3% of composite questions whereas Bing Chat correctly answered 52.4% of questions and Bard correctly answered 51.4% of questions on the OITE. When image-associated questions were excluded, ChatGPT's, Bing Chat's, and Bard's overall accuracies improved to 49.1%, 53.5%, and 56.8%, respectively. Medical students correctly answered 30.8%, and PGY-1, -2, -3, -4, and -5 orthopaedic residents correctly answered 53.1%, 60.4%, 66.6%, 70.0%, and 71.9%, respectively. CONCLUSIONS: ChatGPT, Bard, and Bing Chat are artificial intelligence (AI) models that answered OITE questions with accuracy similar to that of first-year orthopaedic surgery residents. ChatGPT, Bard, and Bing Chat achieved this result without using images or other supplementary media that human test takers are provided. CLINICAL RELEVANCE: Our comparative performance analysis of AI models on orthopaedic board-style questions highlights ChatGPT's, Bing Chat's, and Bard's clinical knowledge and proficiency. Our analysis establishes a baseline of AI model proficiency in the field of orthopaedics and provides a comparative marker for future, more advanced deep learning models. Although in its elementary phase, future AI models' orthopaedic knowledge may provide clinical support and serve as an educational tool.
15
Aidan Gilson, Conrad Safranek, Thomas Huang et al. · JMIR Medical Education · 2023 · 2K citations · Full text
Patrick A. Massey, Carver Montgomery, Andrew S. Zhang · Journal of the American Academy of Orthopaedic Surgeons · 2023 · 194 citations · Full text