Communications in Statistics - Simulation and Computation · 2017 · 22 citations · 0 references
ComputingBiostatisticsComputer ScienceData MiningMachine LearningStatistical ModelingMaximum Likelihood EstimationModel ComparisonModel Fit AssessmentModel SelectionModel ValidationStatistical InferenceCross-validationMachine Learning Prediction ModelsStatistical Learning TheoryStatistical Model SelectionStatistical Model Specification
Generalized degrees of freedom (GDF), as defined by Ye (1998 Ye, J. (1998). On measuring and correcting the effects of data mining and model selection. Journal of the American Statistical Association 93(441):120–131. [Google Scholar] JASA 93:120–131), represent the sensitivity of model fits to perturbations of the data. Such GDF can be computed for any statistical model, making it possible, in principle, to derive the effective number of parameters in machine-learning approaches and thus compute information-theoretical measures of fit. We compare GDF with cross-validation and find that the latter provides a less computer-intensive and more robust alternative. For Bernoulli-distributed data, GDF estimates were unstable and inconsistently sensitive to the number of data points perturbed simultaneously. Cross-validation, in contrast, performs well also for binary data, and for very different machine-learning approaches.