How to Read Articles That Use Machine Learning

TLDR

Recent clinical diagnostic tools increasingly rely on complex machine learning methods, requiring a three‑step evaluation of derivation, validation, and clinical effectiveness, with careful assessment of model type, data appropriateness, and reference standards. Machine learning models include hyperparameters that must be tuned on a dataset separate from the validation set.

Abstract

In recent years, many new clinical diagnostic tools have been developed using complicated machine learning methods. Irrespective of how a diagnostic tool is derived, it must be evaluated using a 3-step process of deriving, validating, and establishing the clinical effectiveness of the tool. Machine learning–based tools should also be assessed for the type of machine learning model used and its appropriateness for the input data type and data set size. Machine learning models also generally have additional prespecified settings called hyperparameters, which must be tuned on a data set independent of the validation set. On the validation set, the outcome against which the model is evaluated is termed the reference standard. The rigor of the reference standard must be assessed, such as against a universally accepted gold standard or expert grading.