Health system-scale language models are all-purpose prediction engines

TLDR

Physicians routinely make time‑constrained decisions, and while structured data‑based predictive models can aid these decisions, their complexity limits everyday use. This study demonstrates that unstructured EHR notes can train a clinical language model that serves as an all‑purpose predictive engine with low‑resistance development and deployment. Using recent NLP advances, the authors trained a large medical language model (NYUTron) and fine‑tuned it for five clinical and operational prediction tasks within their health system. NYUTron achieved AUCs of 78.7–94.9%, outperforming traditional models by 5.36–14.7%, and the study shows that pretraining on clinical text enhances generalizability, supports deployment in a prospective trial, and illustrates the potential for language models to assist physicians at the point of care.

Abstract

Abstract Physicians make critical time-constrained decisions every day. Clinical predictive models can help physicians and administrators make decisions by forecasting clinical and operational events. Existing structured data-based clinical predictive models have limited use in everyday practice owing to complexity in data processing, as well as model development and deployment 1–3 . Here we show that unstructured clinical notes from the electronic health record can enable the training of clinical language models, which can be used as all-purpose clinical predictive engines with low-resistance development and deployment. Our approach leverages recent advances in natural language processing 4,5 to train a large language model for medical language (NYUTron) and subsequently fine-tune it across a wide range of clinical and operational predictive tasks. We evaluated our approach within our health system for five such tasks: 30-day all-cause readmission prediction, in-hospital mortality prediction, comorbidity index prediction, length of stay prediction, and insurance denial prediction. We show that NYUTron has an area under the curve (AUC) of 78.7–94.9%, with an improvement of 5.36–14.7% in the AUC compared with traditional models. We additionally demonstrate the benefits of pretraining with clinical text, the potential for increasing generalizability to different sites through fine-tuning and the full deployment of our system in a prospective, single-arm trial. These results show the potential for using clinical language models in medicine to read alongside physicians and provide guidance at the point of care.