npj Science of Learning · 2024 · 45 citations · 30 references
EducationEducational TechnologyTeachingDiagnostic AssessmentLanguage EducationNatural Language ProcessingConversation AnalysisDialogue AnalysisDialogue SystemsEducational Software EvaluationLanguage ModelingNatural Language Processing In EducationDialog SystemsDialogue EvaluationDialogue Quality EvaluationEvaluation Of Dialogue SystemsInter-coder AgreementInter-coder ReliabilityInter-rater AgreementLarge Language Models
This study explores the use of Large Language Models (LLMs), specifically GPT-4, in analysing classroom dialogue-a key task for teaching diagnosis and quality improvement. Traditional qualitative methods are both knowledge- and labour-intensive. This research investigates the potential of LLMs to streamline and enhance this process. Using datasets from middle school mathematics and Chinese classes, classroom dialogues were manually coded by experts and then analysed with a customised GPT-4 model. The study compares manual annotations with GPT-4 outputs to evaluate efficacy. Metrics include time efficiency, inter-coder agreement, and reliability between human coders and GPT-4. Results show significant time savings and high coding consistency between the model and human coders, with minor discrepancies. These findings highlight the strong potential of LLMs in teaching evaluation and facilitation.
30
The Measurement of Observer Agreement for Categorical Data
J. Richard Landis, Gary G. Koch · Biometrics · 1977 · 78.9K citations · Full text
Advances in neural information processing systems 7
Krzysztof J. Cios, Mark E. Shields · Neurocomputing · 1997 · 22.3K citations
Interrater reliability: the kappa statistic
Marry L. McHugh · Biochemia Medica · 2012 · 18.6K citations · Full text
Interrater reliability: the kappa statistic.
Mary L. McHugh · PubMed · 2012 · 9.2K citations · Full text
On the Dangers of Stochastic Parrots
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major et al. · 2021 · 5.6K citations · Full text
Computing, Computer Science, English +12