OBJECTIVE. Outpatient appointment no-shows are a common problem. Artificial intelligence predictive analytics can potentially facilitate targeted interventions to improve efficiency. We describe a quality improvement project that uses machine learning techniques to predict and reduce outpatient MRI appointment no-shows. MATERIALS AND METHODS. Anonymized records from 32,957 outpatient MRI appointments between 2016 and 2018 were acquired for model training and validation along with a holdout test set of 1080 records from January 2019. The overall no-show rate was 17.4%. A predictive model developed with XGBoost, a decision tree-based ensemble machine learning algorithm that uses a gradient boosting framework, was deployed after various machine learning algorithms were evaluated. The simple intervention measure of using telephone call reminders for patients with the top 25% highest risk of an appointment no-show as predicted by the model was implemented over 6 months. RESULTS. The ROC AUC for the predictive model was 0.746 with an optimized F1 score of 0.708; at this threshold, the precision and recall were 0.606 and 0.852, respectively. The AUC for the holdout test set was 0.738 with an optimized F1 score of 0.721; at this threshold, the precision and recall were 0.605 and 0.893, respectively. The no-show rate 6 months after deployment of the predictive model was 15.9% compared with 19.3% in the preceding 12-month preintervention period, corresponding to a 17.2% improvement from the baseline no-show rate (p < 0.0001). The no-show rates of contactable and noncontactable patients in the group at high risk of appointment no-shows as predicted by the model were 17.5% and 40.3%, respectively (p < 0.0001). CONCLUSION. Machine learning predictive analytics perform moderately well in predicting complex problems involving human behavior using a modest amount of data with basic feature engineering, and they can be incorporated into routine workflow to improve health care delivery.

Seck Guan Foo

Publications