Review of recent systems for automatic assessment of programming assignments

TLDR

Automatic assessment tools for programming exercises offer features such as teacher‑defined tests, resubmission policies, and security measures. The paper systematically reviews recent development of these tools and calls for open collaboration to advance them. It analyzes tool features and pedagogical/technical approaches, and briefly describes selected systems in an appendix. The review identifies emerging features such as web‑software assessment and notes that the proliferation of new systems is driven by current motivations, indicating a need for focused research.

Abstract

This paper presents a systematic literature review of the recent (2006--2010) development of automatic assessment tools for programming exercises. We discuss the major features that the tools support and the different approaches they are using both from the pedagogical and the technical point of view. Examples of these features are ways for the teacher to define tests, resubmission policies, security issues, and so forth. We have also identified a list of novel features, like assessing web software, that are likely to get more research attention in the future. As a conclusion, we state that too many new systems are developed, but also acknowledge the current reasons for the phenomenon. As one solution we encourage opening up the existing systems and joining efforts on developing those further. Selected systems from our survey are briefly described in Appendix A.