Risk-Sensitive Markov Decision Processes of USV Trajectory Planning with Time-Limited Budget

Yi Ding; Hongyang Zhu

doi:10.3390/s23187846

Risk-Sensitive Markov Decision Processes of USV Trajectory Planning with Time-Limited Budget

Sensors (Basel). 2023 Sep 13;23(18):7846. doi: 10.3390/s23187846.

Authors

Yi Ding¹, Hongyang Zhu²

Affiliations

¹ Maritime College, Guangdong Ocean University, Zhanjiang 524091, China.
² College of Mathematics and Computer, Guangdong Ocean University, Zhanjiang 524091, China.

Abstract

Trajectory planning plays a crucial role in ensuring the safe navigation of ships, as it involves complex decision making influenced by various factors. This paper presents a heuristic algorithm, named the Markov decision process Heuristic Algorithm (MHA), for time-optimized avoidance of Unmanned Surface Vehicles (USVs) based on a Risk-Sensitive Markov decision process model. The proposed method utilizes the Risk-Sensitive Markov decision process model to generate a set of states within the USV collision avoidance search space. These states are determined based on the reachable locations and directions considering the time cost associated with the set of actions. By incorporating an enhanced reward function and a constraint time-dependent cost function, the USV can effectively plan practical motion paths that align with its actual time constraints. Experimental results demonstrate that the MHA algorithm enables decision makers to evaluate the trade-off between the budget and the probability of achieving the goal within the given budget. Moreover, the local stochastic optimization criterion assists the agent in selecting collision avoidance paths without significantly increasing the risk of collision.

Keywords: Risk-Sensitive Markov decision processes; trajectory optimization; trajectory planning strategy.

Grants and funding

This research was funded by the Program for the Project of Science and Technology of Zhanjiang City, grant number: 2022B01103, Guangdong Ocean University Teaching Quality Project, grant number: YSZL-201922058-JXD, and the National College Students Innovation and Entrepreneurship Training Program of Guangdong Province, grant number: S202210566074, the National Natural Science Foundation of China, China (grant number: 52171346), the Natural Science Foundation of Guangdong Province, China (grant number: 2021A1515012618), the special projects of key fields (Artificial Intelligence) of Universities in Guangdong Province (grant number: 2019KZDZX1035), China, and program for scientific research start-up funds of Guangdong Ocean University, China.