Automatic control of simulated moving bed process with deep Q-network

Abstract

Optimal control of a simulated moving bed (SMB) process is challenging because the system dynamics is represented as nonlinear partial differential-algebraic equations combined with discrete events. In addition, product purity constraints are active at the optimal operating condition, which implies that these constraints can be easily violated by disturbance. Recently, artificial intelligence techniques have received significant attention for their ability to address complex problems, involving a large number of state variables. In this study, a data-based deep Q-network, which is a model-free reinforcement learning method, is applied to the SMB process to train a near-optimal control policy. Using a deep Q-network, the control policy of a complex dynamic system can be trained off-line as long as a sufficient number of data is provided. These data can be efficiently generated by performing numerical simulations in parallel on multiple machines. The on-line computation of the control input using a trained Q-network is fast enough to satisfy the computational time limit for the SMB process. However, because the Q-network does not predict the future state, it is not possible to explicitly impose state constraints. Instead, the state constraints are indirectly imposed by providing a relatively large penalty (negative reward) when the constraints are violate. Furthermore, logic-based switching control is utilized to limit the ranges of the extract and raffinate purities, which helps to satisfy the state constraints and reduce the regions in the state space for reinforcement learning to explore. The simulation results demonstrate the advantages of applying deep reinforcement learning to control the SMB process.

Keywords: Deep neural network; Optimal control; Reinforcement learning; Simulated moving bed.