From signals to knowledge: A conceptual model for multimodal learning analytics

TLDR

Multimodal learning analytics is gaining attention as sensors and wearable trackers evolve, enabling real‑time capture of observable learner behaviors while latent attributes such as cognition and emotion must still be inferred through human interpretation. The authors performed a literature survey of multimodal learning analytics experiments, examining the input data modalities and the learning theories employed as the hypothesis space. The survey produced the Multimodal Learning Analytics Model, which aims to map multimodal data to improve feedback, integrate machine learning with multimodal inputs, and harmonize terminology between machine learning and learning science.

Abstract

Abstract Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science.