European Journal of Physics · 2024 · 18 citations · 15 references
ComputingComputer ScienceArtificial IntelligenceNatural Language ProcessingText AnalyticsAcademic WritingAuthorship AttributionAutomated Essay ScoringEssay WritingWriting AssessmentAcademic QualityAcademic Writing AssessmentAuthorship Attribution EvaluationComputational Authorship AnalysisComputational Authorship AttributionEssay Quality AssessmentLikert Scale DesignPrecision
Abstract This study aims to compare the academic writing quality and detectability of authorship between human and AI-generated texts by evaluating n = 300 short-form physics essay submissions, equally divided between student work submitted before the introduction of ChatGPT and those generated by OpenAI’s GPT-4. In blinded evaluations conducted by five independent markers who were unaware of the origin of the essays, we observed no statistically significant differences in scores between essays authored by humans and those produced by AI ( p -value = 0.107, α = 0.05). Additionally, when the markers subsequently attempted to identify the authorship of the essays on a 4-point Likert scale—from ‘Definitely AI’ to ‘Definitely Human’—their performance was only marginally better than random chance. This outcome not only underscores the convergence of AI and human authorship quality but also highlights the difficulty of discerning AI-generated content solely through human judgment. Furthermore, the effectiveness of five commercially available software tools for identifying essay authorship was evaluated. Among these, ZeroGPT was the most accurate, achieving a 98% accuracy rate and a precision score of 1.0 when its classifications were reduced to binary outcomes. This result is a source of potential optimism for maintaining assessment integrity. Finally, we propose that texts with ≤50% AI-generated content should be considered the upper limit for classification as human-authored, a boundary inclusive of a future with ubiquitous AI assistance whilst also respecting human-authorship.
15
Fabrizio Dell’Acqua, Edward McFowland, Ethan Mollick et al. · SSRN Electronic Journal · 2023 · 702 citations · Full text
ChatGPT for (Finance) research: The Bananarama Conjecture
Michael Dowling, Brian M. Lucey · Finance research letters · 2023 · 600 citations · Full text
Computing, Data Science, Finance +14
GPT detectors are biased against non-native English writers
Weixin Liang, Mert Yüksekgönül, Yining Mao et al. · Patterns · 2023 · 457 citations · Full text