British Journal of Ophthalmology · 2025 · 17 citations · 22 references
PharmacyOphthalmologyClinical Decision SupportCost-effectiveness AnalysisDiagnostic ReasoningComparative Effectiveness ModelingCost-effectiveness ComparisonCost-effectiveness ModelingDiagnostic Decision SupportHealthcare Cost AnalysisModel-based Economic EvaluationClinical Decision Support ToolsMultiple-choice Question AnsweringOphthalmic DiagnosticsSubspecialty Consultation
BACKGROUND/AIMS: To compare the performance and cost-effectiveness of DeepSeek-R1 with OpenAI o1 in diagnosing and managing ophthalmology clinical cases. METHODS: In this cross-sectional study, a total of 300 clinical cases spanning 10 ophthalmology subspecialties were collected from StatPearls, each with a multiple-choice question on diagnosis or management. DeepSeek-R1 was accessed through its public chat interface, while OpenAI o1 was queried via its Application Programming Interface with a standardised temperature of 0.3. Both models were prompted using plan-and-solve+. Performance was calculated as the proportion of correct answers. McNemar's test was employed to compare the two models' performance on paired data. Intermodel agreement for correct diagnoses was evaluated via Cohen's kappa. Token-based cost analyses were performed to estimate the comparative expenditures of running each model at scale, including input prompts and model-generated output. RESULTS: DeepSeek-R1 and OpenAI o1 achieved an identical overall performance of 82.0% (n=246/300; 95% CI: 77.3 to 85.9). Subspecialty-specific analysis revealed numerical variation in performance, though none of these comparisons reached statistical significance (p>0.05). Agreement in performance between the models was moderate overall (κ=0.503, p<0.001), with substantial agreement in refractive management/intervention (κ=0.698, p<0.001) and moderate agreement in retina/vitreous (κ=0.561, p<0.001) and ocular pathology/oncology (κ=0.495, p<0.01) cases. Cost analysis indicated an approximately 15-fold reduction in per-query, token-related expenses when using DeepSeek-R1 vs OpenAI o1 for the same workload. CONCLUSIONS: DeepSeek-R1 shows strong diagnostic and management performance comparable to OpenAI o1 across ophthalmic subspecialties, while significantly reducing costs. These results support its use as a cost-effective, open-weight alternative to proprietary models.
22
Erik von Elm, Douglas G. Altman, Matthias Egger et al. · Journal of Clinical Epidemiology · 2008 · 16.4K citations · Full text
Public Health, Public Health Practice, Epidemiologic Methods +10
Leakage and the reproducibility crisis in machine-learning-based science
Sayash Kapoor, Arvind Narayanan · Patterns · 2023 · 671 citations · Full text
Evaluating the Performance of ChatGPT in Ophthalmology
Fares Antaki, Samir Touma, Daniel Milad et al. · Ophthalmology Science · 2023 · 494 citations · Full text
John Kounios, Mark Beeman · Current Directions in Psychological Science · 2009 · 424 citations
Nursing, Neuroscience, Psychology +13