A Methodology for Collecting Valid Software Engineering Data

TLDR

The paper presents a data collection method for evaluating software development methodologies and studying the development process. The method uses goal‑directed data collection tied to claimed benefits, defining goals, questions, categorization schemes, and a collection form, gathering change data during development and validating it concurrently through interviews. Applying the method to five projects in two environments demonstrated its feasibility and usefulness, and showed that concurrent validation is essential, as unvalidated data can be up to 50 % erroneous.

Abstract

An effective data collection method for evaluating software development methodologies and for studying the software development process is described. The method uses goal-directed data collection to evaluate methodologies with respect to the claims made for them. Such claims are used as a basis for defining the goals of the data collection, establishing a list of questions of interest to be answered by data analysis, defining a set of data categorization schemes, and designing a data collection form. The data to be collected are based on the changes made to the software during development, and are obtained when the changes are made. To ensure accuracy of the data, validation is performed concurrently with software development and data collection. Validation is based on interviews with those people supplying the data. Results from using the methodology show that data validation is a necessary part of change data collection. Without it, as much as 50 percent of the data may be erroneous. Feasibility of the data collection methodology was demonstrated by applying it to five different projects in two different environments. The application showed that the methodology was both feasible and useful.