Improving the interpretability of deep reinforcement learning in urban drainage system operation

Wenchong Tian; Guangtao Fu; Kunlun Xin; Zhiyu Zhang; Zhenliang Liao

doi:10.1016/j.watres.2023.120912

Improving the interpretability of deep reinforcement learning in urban drainage system operation

Water Res. 2024 Feb 1:249:120912. doi: 10.1016/j.watres.2023.120912. Epub 2023 Nov 23.

Authors

Wenchong Tian¹, Guangtao Fu², Kunlun Xin³, Zhiyu Zhang⁴, Zhenliang Liao⁴

Affiliations

¹ College of Environmental Science and Engineering, Tongji University, Shanghai 200092, PR China; Key Laboratory of Urban Water Supply, Water Saving and Water Environment Governance in the Yangtze River Delta of Ministry of Water Resources, Shanghai 200092, PR China.
² The Centre for Water Systems, University of Exeter, Exeter, UK.
³ College of Environmental Science and Engineering, Tongji University, Shanghai 200092, PR China. Electronic address: xkl@tongji.edu.cn.
⁴ College of Environmental Science and Engineering, Tongji University, Shanghai 200092, PR China.

PMID: 38042066
DOI: 10.1016/j.watres.2023.120912

Abstract

Deep reinforcement learning (DRL) has been increasingly used as an adaptive and efficient solution for real-time control (RTC) of the urban drainage system (UDS). Despite the promising potential of DRL, it is a black-box model whose control logic and control consequences are difficult to be understood and evaluated. This leads to issues of interpretability and poses risks in practical applications. This study develops an evaluation framework to analyze and improve the interpretability of DRL-based UDS operation. The framework includes three analysis methods: Sobol sensitivity analysis, tree-based surrogate modelling, and conditional probability analysis. It is validated using two different DRL approaches, i.e., deep Q-learning network (DQN) and proximal policy optimization (PPO), which are trained to reduce combined sewer overflow (CSO) discharges and flooding in a real-world UDS. According to the results, the two DRLs have been shown to perform better than a rule-based control system that is currently being used. Sobol sensitivity analysis indicates that DQN is particularly sensitive to the flow of links and rainfall, while PPO is sensitive to all the states. Tree-based surrogate models effectively reveal the control logic behind the DRLs and indicate that PPO is more comprehensible but DQN is more forward-looking. Conditional probability analysis demonstrates the potential control consequences of the DRLs and identifies three situations where the DRLs are ineffective: a) the storage of UDS is fully utilized; b) peak flows have already passed through actuators; c) a substantial amount of water enters one location simultaneously. The proposed evaluation framework enhances the interpretability of DRL in UDS operations, fostering trust and confidence from operators, stakeholders, and regulators.

Keywords: Deep reinforcement learning; Interpretability; Sobol sensitivity analysis; Tree-based surrogate model; Urban drainage system.

MeSH terms

Floods*
Probability
Water*

Substances

Water