Mixed Reality for Learning Programming

Abstract

We present our ongoing investigation into leveraging mixed reality (MR) to help students learn coding more easily and with more fun. We have developed an MR coding learning platform using Apple's ARKit 2 on iOS, with a physical user-configurable coding game board. Our approach could provide major benefits over conventional augmented reality (AR) approaches for learning coding and debugging: (1) allowing teachers to tailor the platform to their instructional needs, and spark creativity and engagement among students in designing programming problems that interest them; (2) enabling students to physically interact with a program, concretizing coding errors and providing real-time visual feedback to aid students' program understanding and reduce cognitive load. We present our preliminary results that uses ARKit's image tracking and object detection to enable core mixed-reality interaction capabilities on our platform.