Artificial Intelligence and Public Health: Evaluating ChatGPT Responses to Vaccination Myths and Misconceptions

TLDR

AI tools such as ChatGPT are debated for their potential use in health care contexts. This study assesses the correctness, clarity, and exhaustiveness of ChatGPT’s responses to WHO’s vaccination myths. The authors presented WHO’s 11 vaccination myths to GPT‑3.5 and GPT‑4.0 and had two expert raters evaluate the AI’s answers qualitatively and quantitatively. Raters agreed significantly; overall accuracy was 85.4 %, GPT‑4.0 outperformed GPT‑3.5 in correctness, clarity, and exhaustiveness, and while AI can aid health care when properly queried, it poses risks of misleading non‑experts and raises ethical concerns over the paid version’s higher accuracy.

Abstract

Artificial intelligence (AI) tools, such as ChatGPT, are the subject of intense debate regarding their possible applications in contexts such as health care. This study evaluates the Correctness, Clarity, and Exhaustiveness of the answers provided by ChatGPT on the topic of vaccination. The World Health Organization's 11 "myths and misconceptions" about vaccinations were administered to both the free (GPT-3.5) and paid version (GPT-4.0) of ChatGPT. The AI tool's responses were evaluated qualitatively and quantitatively, in reference to those myth and misconceptions provided by WHO, independently by two expert Raters. The agreement between the Raters was significant for both versions (p of K < 0.05). Overall, ChatGPT responses were easy to understand and 85.4% accurate although one of the questions was misinterpreted. Qualitatively, the GPT-4.0 responses were superior to the GPT-3.5 responses in terms of Correctness, Clarity, and Exhaustiveness (Δ = 5.6%, 17.9%, 9.3%, respectively). The study shows that, if appropriately questioned, AI tools can represent a useful aid in the health care field. However, when consulted by non-expert users, without the support of expert medical advice, these tools are not free from the risk of eliciting misleading responses. Moreover, given the existing social divide in information access, the improved accuracy of answers from the paid version raises further ethical issues.