Multimodal data as a means to understand the learning experience

TLDR

Learning technology research typically relies on click‑stream data to model and predict learner behaviour. The study aims to quantify the advantages of physiological sensing for designing learning technologies. A lab study with 17 participants and 251 game sessions collected click‑stream, eye‑tracking, EEG, video, and wristband data to examine skill development. Fused multimodal data reduced prediction error from 39 % (18 % with feature selection) to 6 %, demonstrating that multimodal physiological sensing substantially improves learning performance predictions and highlights the limitations of click‑stream models.

Abstract

Most work in the design of learning technology uses click-streams as their primary data source for modelling & predicting learning behaviour. In this paper we set out to quantify what, if any, advantages do physiological sensing techniques provide for the design of learning technologies. We conducted a lab study with 251 game sessions and 17 users focusing on skill development (i.e., user's ability to master complex tasks). We collected click-stream data, as well as eye-tracking, electroencephalography (EEG), video, and wristband data during the experiment. Our analysis shows that traditional click-stream models achieve 39% error rate in predicting learning performance (and 18% when we perform feature selection), while for fused multimodal the error drops up to 6%. Our work highlights the limitations of standalone click-stream models, and quantifies the expected benefits of using a variety of multimodal data coming from physiological sensing. Our findings help shape the future of learning technology research by pointing out the substantial benefits of physiological sensing.