Evaluation metrics and statistical tests for machine learning

TLDR

Machine learning research has surged, yet many researchers struggle to evaluate and compare model performance due to statistical unfamiliarity. The paper aims to introduce the most common evaluation metrics for supervised machine‑learning tasks. It explains how to select appropriate statistical tests, gather sufficient metric values, and conduct and interpret the tests. The authors provide practical examples comparing convolutional neural networks for X‑ray lung infection classification and PET tumor detection.

Abstract

Abstract Research on different machine learning (ML) has become incredibly popular during the past few decades. However, for some researchers not familiar with statistics, it might be difficult to understand how to evaluate the performance of ML models and compare them with each other. Here, we introduce the most common evaluation metrics used for the typical supervised ML tasks including binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval. We explain how to choose a suitable statistical test for comparing models, how to obtain enough values of the metric for testing, and how to perform the test and interpret its results. We also present a few practical examples about comparing convolutional neural networks used to classify X-rays with different lung infections and detect cancer tumors in positron emission tomography images.