Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T)

TLDR

Tools can automatically generate unit tests, sometimes achieving higher code coverage than manual tests. The study investigates how effective automatically generated unit tests are at detecting faults in Java programs, using three state‑of‑the‑art tools on 357 real faults from Defects4J. The authors applied Randoop, EvoSuite, and Agitar to the faults, analyzed detection rates, and derived insights for improving coverage, state propagation, and environment simulation. The generated test suites detected 55.7% of faults overall, but only 19.9% of individual suites found any fault.

Abstract

Rather than tediously writing unit tests manually, tools can be used to generate them automatically - sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults. Although the automatically generated test suites detected 55.7% of the faults overall, only 19.9% of all the individual test suites detected a fault. By studying the effectiveness and problems of the individual tools and the tests they generate, we derive insights to support the development of automated unit test generators that achieve a higher fault detection rate. These insights include 1) improving the obtained code coverage so that faulty statements are executed in the first instance, 2) improving the propagation of faulty program states to an observable output, coupled with the generation of more sensitive assertions, and 3) improving the simulation of the execution environment to detect faults that are dependent on external factors such as date and time.