“ChatGPT 4.0 Ghosted Us While Conducting Literature Search:” Modeling the Chatbot’s Generated Non-Existent References Using Regression Analysis

Abstract

The integration of AI technologies like ChatGPT has transformed academic research, yet substantial gaps exist in understanding the implications of AI-generated non-existent references in literature searches. While prior studies have predominantly focused on medical and geography fields using descriptive statistics, a systematic investigation into ChatGPT 4.0’s effectiveness in generating accurate references within the realm of science and technology education remains unexplored, highlighting a significant dearth of research in this critical area. This study, therefore, investigates the reliability of AI-generated references in academic writing utilizing ChatGPT 4.0. Employing a non-experimental correlational design, the research examines the impact of prompt specificity on citation accuracy across various types of prompts, including general, specific, methodological, review, and interdisciplinary prompts. The findings indicate that specific, review, and interdisciplinary prompts correlate positively with accurate references, while general prompts frequently result in non-existent references. Visualizations, including a confusion matrix and precision-recall curve, illustrate the model’s performance. Ultimately, the study underscores the necessity of well-structured prompts to enhance reference quality and cautions against AI-induced hallucinations that produce non-existent references, which can significantly undermine research credibility.