Artificial intelligence (AI) has been widely used in various fields of physics and engineering in recent decades. In this work, we introduce model-based reinforcement learning (MBRL), which is an important branch of machine learning in the AI domain, to the broadband frequency-swept laser control for frequency modulated continuous wave (FMCW) light detection and ranging (LiDAR). With the concern of the direct interaction between the optical system and the MBRL agent, we establish the frequency measurement system model on the basis of the experimental data and the nonlinearity property of the system. In light of the difficulty of this challenging high-dimensional control task, we propose a twin critic network on the basis of the Actor-Critic structure to better learn the complex dynamic characteristics of the frequency-swept process. Furthermore, the proposed MBRL structure would stabilize the optimization process greatly. In the training process of the neural network, we apply a delaying strategy to the policy update and introduce a smoothing regularization strategy to the target policy to further enhance the network stability. With the well-trained control policy, the agent generates the excellent and regularly updated modulation signals to control the laser chirp precisely and an excellent detection resolution is obtained eventually. Our proposed work demonstrates that the integration of data-driven reinforcement learning (RL) and optical system control gives an opportunity to reduce the system complexity and accelerate the investigation and optimization of control systems.