Educational Data Mining and Learning Analytics in Programming

TLDR

Educational data mining and learning analytics promise a deeper understanding of student behavior and knowledge, informing course design, pedagogy, engagement, and risk mitigation. The report surveys the state of educational data mining and learning analytics in programming, introduces a taxonomy for replication studies, reports on three case studies, and outlines future research directions. We describe the current state of collecting and sharing programming data, present a taxonomy for analyzing replication studies, and draw on experiences from three case studies. A literature survey of 2005‑2015 shows a growing number of studies on mining programming processes, yet most rely on simplistic metrics, single institutions, and single courses, underscoring the need for validation, replication, and deeper understanding of contributing factors.

Abstract

Educational data mining and learning analytics promise better understanding of student behavior and knowledge, as well as new information on the tacit factors that contribute to student actions. This knowledge can be used to inform decisions related to course and tool design and pedagogy, and to further engage students and guide those at risk of failure. This working group report provides an overview of the body of knowledge regarding the use of educational data mining and learning analytics focused on the teaching and learning of programming. In a literature survey on mining students' programming processes for 2005-2015, we observe a significant increase in work related to the field. However, the majority of the studies focus on simplistic metric analysis and are conducted within a single institution and a single course. This indicates the existence of further avenues of research and a critical need for validation and replication to better understand the various contributing factors and the reasons why certain results occur. We introduce a novel taxonomy to analyse replicating studies and discuss the importance of replicating and reproducing previous work. We describe what is the state of the art in collecting and sharing programming data. To better understand the challenges involved in replicating or reproducing existing studies, we report our experiences from three case studies using programming data. Finally, we present a discussion of future directions for the education and research community.