BMC Medicine · 2023 · 141 citations · 37 references
ComputingBiostatisticsClinical SciencesClinical Decision SupportClinical InformaticsHealth Data ScienceClinical Decision-makingModel EvaluationRisk PredictionClinical Risk PredictionModel Interpretability In MedicineOptimal Decision MakingPrediction Models
BACKGROUND: Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with "good" or "excellent" models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in "hacking", where researchers are motivated to re-analyse their data until they achieve a "good" result. METHODS: We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline. RESULTS: The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds. CONCLUSIONS: The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.
37
William A. Knaus, Elizabeth A. Draper, Douglas P. Wagner et al. · Critical Care Medicine · 1985 · 13.7K citations
Medicine, Internal Medicine, Clinical Classification Systems +16
APACHE II: a severity of disease classification system.
Elizabeth S. Draper · PubMed · 1985 · 6.1K citations
Karel G.M. Moons, Douglas G. Altman, Johannes B. Reitsma et al. · Annals of Internal Medicine · 2015 · 5.2K citations · Full text
Assessing the Performance of Prediction Models
Ewout W. Steyerberg, Andrew J. Vickers, Nancy R. Cook et al. · Epidemiology · 2009 · 4.9K citations · Full text
Calculating the sample size required for developing a clinical prediction model
Richard D Riley, Joie Ensor, Kym I E Snell et al. · BMJ · 2020 · 2.5K citations · Full text