From Log Files to Assessment Metrics: Measuring Students' Science Inquiry Skills Using Educational Data Mining

TLDR

The study proposes a method to assess students’ science inquiry skill of designing and conducting experiments by mining log data from the Inq‑ITS online microworlds. The approach uses a two‑step process: first, rapid protocol text replay tagging to hand‑score logs, then educational data mining that combines these tags with machine‑derived interaction features to build an automated detector for inquiry performance. The detector, trained on the phase‑change domain, successfully generalizes to the density domain without retraining, demonstrating cross‑domain applicability.

Abstract

We present a method for assessing science inquiry performance, specifically for the inquiry skill of designing and conducting experiments, using educational data mining on students' log data from online microworlds in the Inq-ITS system (Inquiry Intelligent Tutoring System; www.inq-its.org). In our approach, we use a 2-step process: First we use text replay tagging, a type of rapid protocol analysis in which categories are developed and, in turn, used to hand-score students' log data. In the second step, educational data mining is conducted using a combination of the text replay data and machine-distilled features of student interactions in order to produce an automated means of assessing the inquiry skill in question; this is referred to as a detector. Once this detector is appropriately validated, it can be applied to students' log files for auto-assessment and, in the future, to drive scaffolding in real time. Furthermore, we present evidence that this detector developed in 1 scientific domain, phase change, can be used—with no modification or retraining—to effectively detect science inquiry skill in another scientific domain, density.