We propose and numerically investigate a chaotic optical coherent secure communication scheme, which supports long-haul secure transmission for signals in advanced modulation formats. A hybrid optical chaos system is designed with coordination of digital and analog signals. The hybrid entropy source provides a broadband analog optical chaos signal, which could serve as the carrier to load quadrature amplitude modulation (QAM) data. Simultaneously, a digital binary signal generated from the entropy source is transmitted to establish long-haul chaotic synchronization. Coherent detection is utilized at the receiver, and a digital signal processing (DSP) algorithm is adopted to reduce transmission distortion. A 5 Gbaud 16QAM signal is encrypted by a phase chaos carrier with the effective bandwidth of 5.8 GHz. A bit error rate (BER) below forward error correction (FEC) can be achieved after transmitting over 1600 km based on digital-signal-induced chaos synchronization technology. Optimal launch power is investigated to minimize nonlinear effects of transmission links. System security is guaranteed by the high dynamical complexity of the chaotic source and the sensitive time delay as the secret key.