In recent years, the self-homodyne coherent (SHC) system and the constellation shaping (CS) technique have drawn considerable attention due to their abilities to further improve the transmission capacity for various scenarios. From the security point of view, the CS technique and the SHC infrastructure also provide new dimensions for encryption. We propose a high-security and reliable SHC system based on the CS technique and the digital chaos. With a four-dimensional hyperchaotic system, chaotic sequences are generated and used for the exclusive or operation, chaotic constant composition distribution matching, phase disturbance, and optical-layer time-delay disturbance. Moreover, 64-ary circular quadrature amplitude modulation (64CQAM) format is adopted for transmission due to its advantages of sensitivity to phase noise, immunity to conventional digital signal processing, and ability of time-mismatch masking, which is verified by simulation in a SHC system. Last, we conduct an experimental verification in a 20GBaud probabilistically shaped 64CQAM SHC system. Consequently, with a large-linewidth laser source, optical-layer security can be protected by time mismatches of tens of picoseconds. And the digital-layer security is protected by an enormous key space of 10127. The proposed scheme can provide reliable real-time encryption for the optical fiber transmission, serving as a potential candidate for the future high-capacity inter/intra-datacenter security interconnect.