Desirable Characteristics for AI Teaching Assistants in Programming Education

Abstract

Providing timely and personalized feedback to large numbers of students is a\nlong-standing challenge in programming courses. Relying on human teaching\nassistants (TAs) has been extensively studied, revealing a number of potential\nshortcomings. These include inequitable access for students with low confidence\nwhen needing support, as well as situations where TAs provide direct solutions\nwithout helping students to develop their own problem-solving skills. With the\nadvent of powerful large language models (LLMs), digital teaching assistants\nconfigured for programming contexts have emerged as an appealing and scalable\nway to provide instant, equitable, round-the-clock support. Although digital\nTAs can provide a variety of help for programming tasks, from high-level\nproblem solving advice to direct solution generation, the effectiveness of such\ntools depends on their ability to promote meaningful learning experiences. If\nstudents find the guardrails implemented in digital TAs too constraining, or if\nother expectations are not met, they may seek assistance in ways that do not\nhelp them learn. Thus, it is essential to identify the features that students\nbelieve make digital teaching assistants valuable. We deployed an LLM-powered\ndigital assistant in an introductory programming course and collected student\nfeedback ($n=813$) on the characteristics of the tool they perceived to be most\nimportant. Our results highlight that students value such tools for their\nability to provide instant, engaging support, particularly during peak times\nsuch as before assessment deadlines. They also expressed a strong preference\nfor features that enable them to retain autonomy in their learning journey,\nsuch as scaffolding that helps to guide them through problem-solving steps\nrather than simply being shown direct solutions.\n