Chat Generative Pretrained Transformer Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test

Abstract

INTRODUCTION: Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing model that generates human-like text. METHODS: ChatGPT-3 and ChatGPT-4 were used to answer the 2022 and 2021 American College of Gastroenterology self-assessment tests. The exact questions were inputted in both versions of ChatGPT. A score of 70% or higher was required to pass the assessment. RESULTS: Overall, ChatGPT-3 scored 65.1% on 455 included questions and GPT-4 scored 62.4%. DISCUSSION: ChatGPT did not pass the American College of Gastroenterology self-assessment test. We do not recommend its use for medical education in gastroenterology in its current form.