Applicability of non-specification-based approaches to logic testing for software

Abstract

Testing is a crucial part of the development of highly dependable systems. In this paper, we consider the testing of an implementation that is intended to satisfy a Boolean formula. In the literature, specification-based testing has been suggested for this purpose. Typically, such methods first hypothesise a fault class and then generate tests. However, there is almost no research that justifies the fault classes proposed previously. Moreover, the specifications available for automatic test generation are not always available to testers in practice. Based on these observations, we examine the applicability of non-specification-based approaches, which need no specification in the form of a Boolean formula to create tests. We compare a specification-based approach to two non-specification-based approaches, namely random testing and combinatorial testing, which is an emerging technique based on combinatorial designs. The results of an experiment show that combinatorial testing is often comparative to specification-based testing and is always much superior to random testing.