Are automated debugging techniques actually helping programmers?

TLDR

Debugging is notoriously difficult and time‑consuming, and although many automated tools have been developed, they largely rely on strong assumptions and focus on selecting faulty statements, neglecting the complex activities required to understand root causes. The study aims to investigate how developers actually use and benefit from automated debugging tools through human studies. The authors conducted a preliminary study with developers using an automated debugging tool on two tasks, comparing performance with and without the tool. The results show that several assumptions underlying automated debugging techniques are invalid in practice and suggest directions for future research.

Abstract

Debugging is notoriously difficult and extremely time consuming. Researchers have therefore invested a considerable amount of effort in developing automated techniques and tools for supporting various debugging tasks. Although potentially useful, most of these techniques have yet to demonstrate their practical effectiveness. One common limitation of existing approaches, for instance, is their reliance on a set of strong assumptions on how developers behave when debugging (e.g., the fact that examining a faulty statement in isolation is enough for a developer to understand and fix the corresponding bug). In more general terms, most existing techniques just focus on selecting subsets of potentially faulty statements and ranking them according to some criterion. By doing so, they ignore the fact that understanding the root cause of a failure typically involves complex activities, such as navigating program dependencies and rerunning the program with different inputs. The overall goal of this research is to investigate how developers use and benefit from automated debugging tools through a set of human studies. As a first step in this direction, we perform a preliminary study on a set of developers by providing them with an automated debugging tool and two tasks to be performed with and without the tool. Our results provide initial evidence that several assumptions made by automated debugging techniques do not hold in practice. Through an analysis of the results, we also provide insights on potential directions for future work in the area of automated debugging.