Intelligent tutoring systems and learning outcomes: A meta-analysis.

TLDR

Intelligent Tutoring Systems are computer programs that model learners’ psychological states to deliver individualized instruction across diverse subjects such as algebra, medicine, law, and reading. A meta‑analysis of 107 effect sizes from 14,321 participants examined how ITS outcomes compared to non‑ITS environments, varying by ITS type, comparison treatment, learning outcome, and knowledge type. ITS produced significantly higher achievement than teacher‑led large‑group instruction (g .42), non‑ITS computer‑based instruction (g .57), and textbooks (g .35), but did not differ from individualized human tutoring (g –.11) or small‑group instruction (g .05), with positive effects across all education levels, subject domains, and instructional roles, and no evidence of publication bias.

Abstract

Intelligent Tutoring Systems (ITS) are computer programs that model learners’ psychological states to provide individualized instruction. They have been developed for diverse subject areas (e.g., algebra, medicine, law, reading) to help learners acquire domain-specific, cognitive and metacognitive knowledge. A meta-analysis was conducted on research that compared the outcomes from students learning from ITS to those learning from non-ITS learning environments. The meta-analysis examined how effect sizes varied with type of ITS, type of comparison treatment received by learners, type of learning outcome, whether knowledge to be learned was procedural or declarative, and other factors. After a search of major bibliographic databases, 107 effect sizes involving 14,321 participants were extracted and analyzed. The use of ITS was associated with greater achievement in comparison with teacher-led, large-group instruction (g .42), non-ITS computer-based instruction (g .57), and textbooks or workbooks (g .35). There was no significant difference between learning from ITS and learning from individualized human tutoring (g –.11) or small-group instruction (g .05). Significant, positive mean effect sizes were found regardless of whether the ITS was used as the principal means of instruction, a supplement to teacher-led instruction, an integral component of teacher-led instruction, or an aid to homework. Significant, positive effect sizes were found at all levels of education, in almost all subject domains evaluated, and whether or not the ITS provided feedback or modeled student misconceptions. The claim that ITS are relatively effective tools for learning is consistent with our analysis of potential publication bias.