Subcutaneous insulin administration by deep reinforcement learning for blood glucose level control of type-2 diabetic patients

Abstract

Background: Type-2 diabetes mellitus is characterized by insulin resistance and impaired insulin secretion in the human body. Many endeavors have been made in terms of controlling and reducing blood glucose via the medium of automated controlling tools to increase precision and efficiency and reduce human error. Recently, reinforcement learning algorithms are proved to be powerful in the field of intelligent control, which was the motivation for the current study.

Methods: For the first time, a reinforcement algorithm called normalized advantage function (NAF) algorithm has been applied as a model-free reinforcement learning method to regulate the blood glucose level of type-2 diabetic patients through subcutaneous injection. The algorithm has been designed and developed in a model-free approach to avoid additional inaccuracies and parameter uncertainty introduced by the mathematical models of the glucoregulatory system. Insulin doses constitute the control action that is designed to be stated directly in clinical language with the unit IU. In this regard, a new environment state is considered in addition to the glucose level to take into account the delayed effect of insulin elimination under the skin. Finally, a simple but practical reward function is developed to be used with the NAF algorithm to correct the glucose level and maintain it in the desired range.

Results: The simulation environment was set up to imitate the basal-bolus process accurately. Results for 30 days of simulation of the designed controller on three different average virtual patients verify the feasibility and effectiveness of the method and reveal our proposed controller's learning ability. Moreover, as the insulin elimination dynamic was taken into account, a more complete and more realistic model than the previously studied models has emerged.

Conclusion: NAF has proved a promising control approach, able to successfully regulate and significantly reduce the fluctuation of the blood glucose without meal announcements, compared to standard optimized open-loop basal-bolus therapies. The method and its results, which are directly in the clinical language, are applicable in real-time clinical situations.

Keywords: Insulin administration; Normalized advantage function; Subcutaneous injection; Type-2 diabetes.