Using Networks to Visualize and Analyze Process Data for Educational Assessment

TLDR

Interactive, adaptive scenario‑based tasks are enabled by new technology, but their complex data pose challenges for building suitable psychometric models. The study investigates process data from scenario‑based tasks, examining how network measures relate to scoring rubrics and enable intergroup comparisons. Using social network analysis, the authors constructed weighted directed transition networks from empirical process data, where nodes represent actions and links connect successive actions in the sequence. Visualization of the transition networks reveals process data patterns that offer insights for item design.

Abstract

New technology enables interactive and adaptive scenario‐based tasks (SBTs) to be adopted in educational measurement. At the same time, it is a challenging problem to build appropriate psychometric models to analyze data collected from these tasks, due to the complexity of the data. This study focuses on process data collected from SBTs. We explore the potential of using concepts and methods from social network analysis to represent and analyze process data. Empirical data were collected from the assessment of Technology and Engineering Literacy, conducted as part of the National Assessment of Educational Progress. For the activity sequences in the process data, we created a transition network using weighted directed networks, with nodes representing actions and directed links connecting two actions only if the first action is followed by the second action in the sequence. This study shows how visualization of the transition networks represents process data and provides insights for item design. This study also explores how network measures are related to existing scoring rubrics and how detailed network measures can be used to make intergroup comparisons.