Leveraging multimodal learning analytics to differentiate student learning strategies

TLDR

Multimodal analysis has proven effective in modeling human interactions, and prior work shows that the experimental conditions used in this study are linked to differences in learning gains and design quality. The study aims to identify behavioral practices that differ between two experimental conditions to better understand learning interventions, and to demonstrate how multimodal learning analytics can be applied in complex engineering design contexts. We compare uni‑modal and multimodal, manual and semi‑automated methods—including human annotations, speech, gesture, and electro‑dermal activation data—from a study of 20 students in two experimental conditions, illustrating how the same algorithm applied to different data forms yields complementary insights.

Abstract

Multimodal analysis has had demonstrated effectiveness in studying and modeling several human-human and human-computer interactions. In this paper, we explore the role of multimodal analysis in the service of studying complex learning environments. We compare uni-modal and multimodal; manual and semi-automated methods for examining how students learn in a hands-on, engineering design context. Specifically, we compare human annotations, speech, gesture and electro-dermal activation data from a study (N=20) where student participating in two different experimental conditions. The experimental conditions have already been shown to be associated with differences in learning gains and design quality. Hence, one objective of this paper is to identify the behavioral practices that differed between the two experimental conditions, as this may help us better understand how the learning interventions work. An additional objective is to provide examples of how to conduct learning analytics research in complex environments and compare how the same algorithm, when used with different forms of data can provide complementary results.