Standard vs. non-standard cross-validation: evaluation of performance in a space with structured distribution of datapoints

Abstract

Cross-validation is a popularly used approach to evaluation of performance for classifiers. It relies on random selection of independent samples for training and testing, and assumes that if any similarities among samples exist, they do not lead to known grouping of datapoints in the input space. If these conditions are violated, as it may happen for datasets with some structure of samples included, standard cross-validation can return biased results even for many folds. In the paper the research on cross-validation was reported for application to stylometric datasets, describing a task of authorship attribution. The comparison of standard and non-standard processing was presented. In the latter case, selected subsets of examples were swapped over between training and test sets several times. The experiments with three popular classifiers showed that standard cross-validation tended to give over-optimistic results, whereas non-standard processing was more guarded, and by that more reliable. To avoid high computational costs involved, evaluation based on averaged predictions for limited numbers of test sets can be considered as a reasonable compromise.