PMLB: a large benchmark suite for machine learning evaluation and comparison

TLDR

Machine‑learning method selection is challenging, and although many benchmark datasets exist, their inconsistent organization hampers standardization. The study introduces an accessible, curated, and evolving public benchmark suite to ease the identification of strengths and weaknesses across machine‑learning methods. The authors compare meta‑features of the suite’s datasets to assess diversity, then apply established algorithms to the entire benchmark and analyze performance clustering. The analysis reveals that current benchmarks lack sufficient diversity, leaving gaps that the new resource aims to fill, thereby advancing understanding of benchmarking limitations.

Abstract

The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different sources, but their organization and adoption as standards have been inconsistent. As such, selecting and curating specific benchmarks remains an unnecessary burden on machine learning practitioners and data scientists.The present study introduces an accessible, curated, and developing public benchmark resource to facilitate identification of the strengths and weaknesses of different machine learning methodologies. We compare meta-features among the current set of benchmark datasets in this resource to characterize the diversity of available data. Finally, we apply a number of established machine learning methods to the entire benchmark suite and analyze how datasets and algorithms cluster in terms of performance. From this study, we find that existing benchmarks lack the diversity to properly benchmark machine learning algorithms, and there are several gaps in benchmarking problems that still need to be considered.This work represents another important step towards understanding the limitations of popular benchmarking suites and developing a resource that connects existing benchmarking standards to more diverse and efficient standards in the future.