A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

TLDR

Formative feedback is crucial for learning, and many programming exercise tools provide automated feedback to students. We conducted a systematic literature review, developed a labeling scheme, and applied Narciss’ feedback content categories to classify the tools and their feedback messages. The review of 101 tools revealed that most automated feedback merely identifies mistakes, offers little guidance for improvement, is difficult for teachers to adapt, yet the diversity of feedback types and generation techniques has expanded over recent decades.

Abstract

Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.