Experiments with a new boosting algorithm

TLDR

AdaBoost is a boosting algorithm that can markedly reduce the error of weak learners, and a pseudo‑loss technique was introduced to concentrate learning on the hardest labels. This study evaluates AdaBoost, with and without pseudo‑loss, on real‑world learning tasks. The authors conducted two experimental series: first comparing AdaBoost to Breiman's bagging across multiple classifiers on benchmark datasets, and second examining AdaBoost with a nearest‑neighbor classifier on an OCR task.

Abstract

In an earlier paper, we introduced a new algorithm called AdaBoost which, theoretically, can be used to significantly reduce the error of any learning algorithm that con- sistently generates classifiers whose performance is a little better than random guessing. We also introduced the related notion of a pseudo-loss which is a method for forcing a learning algorithm of multi-label concepts to concentrate on the labels that are hardest to discriminate. In this paper, we describe experiments we carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems. We performed two sets of experiments. The first set compared boosting to Breiman's bagging method when used to aggregate various classifiers (including decision trees and single attribute- value tests). We compared the performance of the two methods on a collection of machine-learning benchmarks. In the second set of experiments, we studied in more detail the performance of boosting using a nearest-neighbor classifier on an OCR problem.