Cause reduction: delta debugging, even without bugs

TLDR

Test cases are used to expose faults, exercise code, or produce desired output, and delta debugging removes accidental parts to produce smaller, easier‑to‑understand test cases. This paper extends delta debugging by introducing cause reduction, a method that simplifies test cases relative to arbitrary effects. Cause reduction generalizes delta debugging by iteratively removing test case components that are not essential for the specified effect, producing a minimal test case. Cause‑reduced test suites proved effective as quick tests for real‑world programs, outperforming Mozilla’s JavaScript engine suite, maintaining effectiveness for up to four months, improving branch coverage by over 500, and doubling branches explored in seeded symbolic execution. © 2015 John Wiley & Sons, Ltd.

Abstract

Summary What is a test case for ? Sometimes, to expose a fault. Tests can also exercise code, use memory or time, or produce desired output. Given a desired effect, a test case can be seen as a cause , and its components divided into essential (required for effect) and accidental. Delta debugging is used for removing accidents from failing test cases, producing smaller test cases that are easier to understand. This paper extends delta debugging by simplifying test cases with respect to arbitrary effects , a generalization called cause reduction . Suites produced by cause reduction provide effective quick tests for real‐world programs. For Mozilla's JavaScript engine, the reduced suite is possibly more effective for finding faults. The effectiveness of reduction‐based suites persists through changes to the software, improving coverage by over 500 branches for versions up to 4 months later. Cause reduction has other applications, including improving seeded symbolic execution, where using reduced tests can often double the number of additional branches explored. Copyright © 2015 John Wiley & Sons, Ltd.