Building knowledge through families of experiments

TLDR

Software engineering experimentation is essential yet challenging due to numerous context variables, requiring a community-driven framework to motivate studies, integrate results, and build shared models. The authors aim to present a framework for organizing related studies based on their experimental experience. The framework is supported by the authors’ empirical work, detailing persistent design problems, validity threats, evaluation criteria, and execution practices in software engineering experiments. The framework enables viewing experiments as families, generating novel hypotheses and supporting incremental knowledge building through replication.

Abstract

Experimentation in software engineering is necessary but difficult. One reason is that there are a large number of context variables and, so, creating a cohesive understanding of experimental results requires a mechanism for motivating studies and integrating results. It requires a community of researchers that can replicate studies, vary context variables, and build models that represent the common observations about the discipline. The paper discusses the experience of the authors, based upon a collection of experiments, in terms of a framework for organizing sets of related studies. With such a framework, experiments can be viewed as part of common families of studies, rather than being isolated events. Common families of studies can contribute to important and relevant hypotheses that may not be suggested by individual experiments. A framework also facilitates building knowledge in an incremental manner through the replication of experiments within families of studies. To support the framework, the paper discusses the experiences of the authors in carrying out empirical studies, with specific emphasis on persistent problems encountered in experimental design, threats to validity, criteria for evaluation, and execution of experiments in the domain of software engineering.