Error Correlation and Error Reduction in Ensemble Classifiers

TLDR

Ensemble classifiers can improve generalization, but the gains depend more on the selection of data presented to the combiner than on the combining method itself. This study investigates data selection and classifier training techniques to prepare classifiers for effective combination. The authors review a framework that quantifies the need to reduce inter‑classifier correlation and discuss methods to make ensemble members more complementary. Experiments demonstrate that reducing correlation among classifiers yields benefits and pitfalls, particularly when training data are scarce. Keywords: combining, cross‑validation, error correlation, error reduction, ensemble classifiers, bootstrapping, resampling.

Abstract

Abstract Using an ensemble of classifiers, instead of a single classifier, can lead to improved generalization. The gains obtained by combining, however, are often affected more by the selection of what is presented to the combiner than by the actual combining method that is chosen. In this paper, we focus on data selection and classifier training methods, in order to 'prepare' classifiers for combining. We review a combining framework for classification problems that quantifies the need for reducing the correlation among individual classifiers. Then, we discuss several methods that make the classifiers in an ensemble more complementary. Experimental results are provided to illustrate the benefits and pitfalls of reducing the correlation among classifiers, especially when the training data are in limited supply. Keywords: CombiningCross-validationError CorrelationError ReductionEnsemble ClassifiersBootstrappingResampling