2002 · 381 citations · 17 references
EducationDecision SciencesExperimental PerformancePerformance MeasurementTest DevelopmentComputer-based TestingSoftware TestingPrioritizationRegression Test PrioritizationRegression Test SelectionSoftware Maintenance ProcessTest Case PrioritizationTest CasesTest PrioritizationTest Selection
Regression testing is costly and frequently executed, and while regression test selection techniques aim to reduce costs, they often ignore practical resource constraints, leading testers to prioritize tests and violate RTS assumptions, which undermines performance guarantees. The authors propose a history‑based test prioritization technique that uses past execution data to guide test ordering. They evaluated this technique by conducting an experiment measuring its impact on long‑run performance in resource‑constrained regression testing scenarios. Results reveal key trade‑offs that must be considered when applying the technique across multiple software releases.
Regression testing is an expensive and frequently executed maintenance process used to revalidate modified software. To improve it, regression test selection (RTS) techniques strive to lower costs without overly reducing effectiveness by carefully selecting a subset of the test suite. Under certain conditions, some can even guarantee that the selected test cases perform no worse than the original test suite.But this ignores certain software development realities such as resource and time constraints that may prevent using RTS techniques as intended (e.g., regression testing must be done overnight, but RTS selection returns two days worth of tests). In practice, testers work around this by prioritizing the test cases and running only those that fit within existing constraints. Unfortunately this generally violates key RTS assumptions, voiding RTS technique guarantees and making regression testing performance unpredictable.Despite this, existing prioritization techniques are memoryless, implicitly assuming that local choices can ensure adequate long run performance. Instead, we proposed a new technique that bases prioritization on historical execution data. We conducted an experiment to assess its effects on the long run performance of resource constrained regression testing. Our results expose essential tradeoffs that should be considered when using these techniques over a series of software releases.
17