Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation

TLDR

In machine learning, classifier performance is typically assessed by prediction error, which must be estimated in real‑world settings, making the choice of an appropriate error estimator critical. This study investigates the statistical bias and variance of the kappa‑fold cross‑validation error estimator, aiming to understand its properties. The authors provide a novel variance decomposition that separates sensitivity to training‑set changes from fold changes, compare bias and variance across kappa values, and validate the theory with experiments on naive Bayes and nearest‑neighbor classifiers over varied folds, sample sizes, and data distributions in artificial domains. The paper offers practical recommendations for selecting kappa in cross‑validation to optimize error estimation.

Abstract

In the machine learning field, the performance of a classifier is usually measured in terms of prediction error. In most real-world problems, the error cannot be exactly calculated and it must be estimated. Therefore, it is important to choose an appropriate estimator of the error. This paper analyzes the statistical properties, bias and variance, of the kappa-fold cross-validation classification error estimator (kappa-cv). Our main contribution is a novel theoretical decomposition of the variance of the kappa-cv considering its sources of variance: sensitivity to changes in the training set and sensitivity to changes in the folds. The paper also compares the bias and variance of the estimator for different values of kappa. The experimental study has been performed in artificial domains because they allow the exact computation of the implied quantities and we can rigorously specify the conditions of experimentation. The experimentation has been performed for two classifiers (naive Bayes and nearest neighbor), different numbers of folds, sample sizes, and training sets coming from assorted probability distributions. We conclude by including some practical recommendation on the use of kappa-fold cross validation.