Continuous variable quantum key distribution (CV-QKD) can guarantee that two parties share secure keys even in the presence of an eavesdropper. However, the polarization direction of the coherent state transmitted in CV-QKD is susceptible to environmental disturbances during channel transmission, making it difficult to share keys consistently over long periods of time. Therefore, a CV-QKD system that can resist environmental disturbance is very urgent. In this paper, we propose a new optical architecture for CV-QKD based on the Faraday-Michelson interference (FMI) structure, and finally form an all-single-mode (SM) fiber-based stable CV-QKD system which employs transmitted local oscillator (TLO) scheme and discrete modulation coherent state (DMCS) protocol. Specifically, since the Faraday mirror rotates the polarization direction of light by 90o, the birefringence effect of light can be effectively dealt with, thus ensuring the same polarization state of light before and after reflection. The final simulation results show that the theoretical secret key rate of this scheme can reach 139 kbps at 70 km, which can further improve the stability and robustness of CV-QKD in the real environment, and provide technical support for the next-generation high-stability QKD system.