EVALUATION OF CLASSIFICATION ALGORITHMS USING MCDM AND RANK CORRELATION

TLDR

Classification algorithm selection is critical across disciplines and is typically framed as a multiple‑criteria decision‑making problem, yet different MCDM methods often yield divergent classifier rankings. This study proposes a method that uses Spearman’s rank correlation to reconcile disagreements among MCDM approaches to classifier ranking. The authors evaluate five MCDM techniques on 17 classifiers, 10 performance metrics, and 11 public binary‑classification datasets. Experimental results show that the proposed method substantially reduces ranking discrepancies, achieving consensus among the MCDM methods.

Abstract

Classification algorithm selection is an important issue in many disciplines. Since it normally involves more than one criterion, the task of algorithm selection can be modeled as multiple criteria decision making (MCDM) problems. Different MCDM methods evaluate classifiers from different aspects and thus they may produce divergent rankings of classifiers. The goal of this paper is to propose an approach to resolve disagreements among MCDM methods based on Spearman's rank correlation coefficient. Five MCDM methods are examined using 17 classification algorithms and 10 performance criteria over 11 public-domain binary classification datasets in the experimental study. The rankings of classifiers are quite different at first. After applying the proposed approach, the differences among MCDM rankings are largely reduced. The experimental results prove that the proposed approach can resolve conflicting MCDM rankings and reach an agreement among different MCDM methods.