NEJM AI · 2024 · 128 citations · 14 references
MedicineInternal MedicinePediatricsArtificial IntelligenceNatural Language ProcessingResidency TrainingClinical Natural Language ProcessingMedical Technology AssessmentResident AssessmentSurgical ResidencyBenchmark Dataset DesignBenchmark Dataset EvaluationLarge Language ModelsMedical SpecializationModel Performance AnalysisModel Performance EvaluationPhysician Performance MeasurementTransformer Models
BACKGROUND Artificial intelligence (AI) is a burgeoning technological advancement, with considerable promise for influencing the field of medicine. As a preliminary step toward integrating AI into medical practice, it is imperative to ascertain whether model performance is comparable with that of physicians. We present a systematic comparison of performance by a large language model (LLM) versus that of a large cohort of physicians. The cohort includes all residents who took the medical specialist license examination in Israel in 2022 across the core medical disciplines: internal medicine, general surgery, pediatrics, psychiatry, and obstetrics and gynecology (OB/GYN). We provide the examinations as an accessible benchmark dataset for the medical machine learning and natural language processing communities, which may be adapted for future LLM studies.
14
Harrison's Principles of Internal Medicine.
Annals of Internal Medicine · 1988 · 12.6K citations
Large language models encode clinical knowledge
Karan Singhal, Shekoofeh Azizi, Tao Tu et al. · Nature · 2023 · 3.2K citations · Full text
Computing, Health Informatics, Computational Linguistics +16
Aidan Gilson, Conrad Safranek, Thomas Huang et al. · JMIR Medical Education · 2023 · 2K citations · Full text
Kristina Lång, Viktoria Josefsson, Anna-Maria Larsson et al. · The Lancet Oncology · 2023 · 509 citations · Full text