Novel data-driven two-dimensional Q-learning for optimal tracking control of batch process with unknown dynamics

Xin Wen; Huiyuan Shi; Chengli Su; Xueying Jiang; Ping Li; Jingxian Yu

doi:10.1016/j.isatra.2021.06.007

Novel data-driven two-dimensional Q-learning for optimal tracking control of batch process with unknown dynamics

ISA Trans. 2022 Jun:125:10-21. doi: 10.1016/j.isatra.2021.06.007. Epub 2021 Jun 7.

Authors

Xin Wen¹, Huiyuan Shi², Chengli Su³, Xueying Jiang⁴, Ping Li⁵, Jingxian Yu⁶

Affiliations

¹ School of Information and Control Engineering, Liaoning Petrochemical University, China.
² School of Information and Control Engineering, Liaoning Petrochemical University, China; School of Automation, Northwestern Polytechnical University, China; State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang, China. Electronic address: huiyuan_shi@lnpu.edu.cn.
³ School of Information and Control Engineering, Liaoning Petrochemical University, China; School of Electronic and Information Engineering, University of Science and Technology Liaoning, China. Electronic address: sclwind@sina.com.
⁴ School of Information Science and Engineering, Northeastern University, China.
⁵ School of Information and Control Engineering, Liaoning Petrochemical University, China; School of Electronic and Information Engineering, University of Science and Technology Liaoning, China.
⁶ School of Sciences, Liaoning Petrochemical University, China.

PMID: 34130858
DOI: 10.1016/j.isatra.2021.06.007

Abstract

In view that the previous control methods usually rely too much on the models of batch process and have difficulty in a practical batch process with unknown dynamics, a novel data-driven two-dimensional (2D) off-policy Q-learning approach for optimal tracking control (OTC) is proposed to make the batch process obtain a model-free control law. Firstly, an extended state space equation composing of the state and output error is established for ensuring tracking performance of the designed controller. Secondly, the behavior policy of generating data and the target policy of optimization as well as learning is introduced based on this extended system. Then, the Bellman equation independent of model parameters is given via analyzing the relation between 2D value function and 2D Q-function. The measured data along the batch and time directions of batch process are just taken to carry out the policy iteration, which can figure out the optimal control problem despite lacking systematic dynamic information. The unbiasedness and convergence of the designed 2D off-policy Q-learning algorithm are proved. Finally, a simulation case for injection molding process manifests that control effect and tracking effect gradually become better with the increasing number of batches.

Keywords: 2D off-policy Q-learning; Batch process; Data-driven; Injection molding; Optimal tracking control.