An empirical investigation of the relationship between spectra differences and regression faults

TLDR

Many software maintenance and testing tasks involve comparing the behaviours of program versions, and program spectra have recently been proposed as a heuristic for such comparisons. The study aims to assess the usefulness of spectra by examining the relationship between spectra differences and regression faults, and to empirically compare several spectra types. An experiment was conducted that measured spectra differences and regression fault exposure across program versions. Certain spectra differences correlate strongly with regression fault exposure in one direction, but not vice versa, and more precise spectra types do not yield better practical precision than cheaper alternatives, implying limited practical benefit. Copyright © 2000 John Wiley & Sons, Ltd.

Abstract

Many software maintenance and testing tasks involve comparing the behaviours of program versions. Program spectra have recently been proposed as a heuristic for use in performing such comparisons. To assess the potential usefulness of spectra in this context an experiment was conducted, examining the relationship between differences in program spectra and the exposure of regression faults (faults existing in a modified version of a program that were not present prior to modifications, or not revealed in previous testing), and empirically comparing several types of spectra. The results reveal that certain types of spectra differences correlate with high frequency—at least in one direction—with the exposure of regression faults. That is, when regression faults are revealed by particular inputs, spectra differences are likely also to be revealed by those inputs, though the reverse is not true. The results also suggest that several types of spectra that appear, analytically, to offer greater precision in predicting the presence of regression faults than other, cheaper, spectra may provide no greater precision in practice. These results have ramifications for future research on, and for the practical uses of, program spectra. Copyright © 2000 John Wiley & Sons, Ltd.