JMIR mhealth and uhealth · 2024 · 31 citations · 14 references
ComputingHealth InformaticsArtificial IntelligenceExperimental DesignMachine LearningClinical TrialsGenerative ModelingGenerative EvaluationMethod ReproducibilityReproducibility Assessment
The increasing interest in the potential applications of generative artificial intelligence (AI) models like ChatGPT in health care has prompted numerous studies to explore its performance in various medical contexts. However, evaluating ChatGPT poses unique challenges due to the inherent randomness in its responses. Unlike traditional AI models, ChatGPT generates different responses for the same input, making it imperative to assess its stability through repetition. This commentary highlights the importance of including repetition in the evaluation of ChatGPT to ensure the reliability of conclusions drawn from its performance. Similar to biological experiments, which often require multiple repetitions for validity, we argue that assessing generative AI models like ChatGPT demands a similar approach. Failure to acknowledge the impact of repetition can lead to biased conclusions and undermine the credibility of research findings. We urge researchers to incorporate appropriate repetition in their studies from the outset and transparently report their methods to enhance the robustness and reproducibility of findings in this rapidly evolving field.
14
Aidan Gilson, Conrad Safranek, Thomas Huang et al. · JMIR Medical Education · 2023 · 2K citations · Full text
ChatGPT: the future of discharge summaries?
Sajan B Patel, Kyle Lam · The Lancet Digital Health · 2023 · 734 citations · Full text
Using ChatGPT to write patient clinic letters
Stephen R Ali, Thomas D. Dobbs, Hayley Hutchings et al. · The Lancet Digital Health · 2023 · 387 citations · Full text
The exciting potential for ChatGPT in obstetrics and gynecology
Amos Grünebaum, Joseph Chervenak, Susan L. Pollet et al. · American Journal of Obstetrics and Gynecology · 2023 · 234 citations · Full text
Medicine, Gynecology, Obstetrics +11