Test case prioritization: a family of empirical studies

TLDR

Test case prioritization reduces regression testing cost by running more important tests earlier and has been shown to improve fault detection rates. This work investigates whether prioritization is effective on specific modified versions, the trade‑offs between fine‑ and coarse‑granularity techniques, and whether adding fault‑proneness metrics enhances effectiveness. We conducted controlled experiments and case studies that empirically compared various prioritization techniques to answer these questions.

Abstract

To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of such prioritization is to increase a test suite's rate of fault detection. Previous work reported results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: 1) Can prioritization techniques be effective when targeted at specific modified versions; 2) what trade-offs exist between fine granularity and coarse granularity prioritization techniques; 3) can the incorporation of measures of fault proneness into prioritization techniques improve their effectiveness? To address these questions, we have performed several new studies in which we empirically compared prioritization techniques using both controlled experiments and case studies.