Hybrid Control Policy for Artificial Pancreas via Ensemble Deep Reinforcement Learning

Abstract

OBJECTIVE: The artificial pancreas (AP) shows promise for closed-loop glucose control in type 1 diabetes mellitus (T1DM). However, designing effective control policies for the AP remains challenging due to complex physiological processes, delayed insulin response, and inaccurate glucose measurements. While model predictive control (MPC) offers safety and stability through the dynamic model and safety constraints, it lacks individualization and is adversely affected by unannounced meals. Conversely, deep reinforcement learning (DRL) provides personalized and adaptive strategies but struggles with distribution shifts and substantial data requirements. METHODS: We propose a hybrid control policy for the artificial pancreas (HyCPAP) to address the above challenges. HyCPAP combines an MPC policy with an ensemble DRL policy, leveraging the strengths of both policies while compensating for their respective limitations. To facilitate faster deployment of AP systems in real-world settings, we further incorporate meta-learning techniques into HyCPAP, leveraging previous experience and patient-shared knowledge to enable fast adaptation to new patients with limited available data. RESULTS: We conduct extensive experiments using the UVA/Padova T1DM simulator across five scenarios. Our approaches achieve the highest percentage of time spent in the desired range and the lowest occurrences of hypoglycemia. CONCLUSION: The results clearly demonstrate the superiority of our methods for closed-loop glucose management in individuals with T1DM. SIGNIFICANCE: The study presents novel control policies for AP systems, affirming their great potential for efficient closed-loop glucose control.