IEEE Transactions on Software Engineering · 2012 · 241 citations · 39 references
ManagementComputer ScienceProject ManagementCost ManagementCost-benefit AnalysisSoftware EngineeringCost AnalysisCost EstimationCost MeasurementProject Cost ManagementSoftware MetricsCost Estimation MethodologyEffort EstimationEffort Estimation ModelsExperimental StudiesPrediction ModelsProject EstimationSoftware Estimation
Software Cost Estimation can be described as the process of predicting the most realistic effort required to complete a software project. Due to the strong relationship of accurate effort estimations with many crucial project management activities, the research community has been focused on the development and application of a vast variety of methods and models trying to improve the estimation procedure. From the diversity of methods emerged the need for comparisons to determine the best model. However, the inconsistent results brought to light significant doubts and uncertainty about the appropriateness of the comparison process in experimental studies. Overall, there exist several potential sources of bias that have to be considered in order to reinforce the confidence of experiments. In this paper, we propose a statistical framework based on a multiple comparisons algorithm in order to rank several cost estimation models, identifying those which have significant differences in accuracy, and clustering them in nonoverlapping groups. The proposed framework is applied in a large-scale setup of comparing 11 prediction models over six datasets. The results illustrate the benefits and the significant information obtained through the systematic comparison of alternative methods.
39
Statistical Comparisons of Classifiers over Multiple Data Sets
Janez Demšar · 2006 · 11.2K citations
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Ron Kohavi · 1995 · 10.7K citations · Full text
A Cluster Analysis Method for Grouping Means in the Analysis of Variance
A.J. Scott, M. Knott · Biometrics · 1974 · 2.8K citations
Design and Analysis of Experiments
Syamala Srinivasan, Douglas C. Montgomery · Technometrics · 1992 · 2.6K citations