We propose and experimentally study a coherent optical secure transmission system based on one dual-polarization in-phase and quadrature modulator (IQM). One beam of the polarized light is used to generate broadband chaos by configuring a nonlinear opto-electronic oscillator while the other beam carries the encrypted signal. The encrypted signal is obtained through sequential encryption of the analog and digital chaos. The mutual mask of the hybrid chaotic signals can effectively enhance the security performance. Moreover, by varying the encryption depth of analog and digital vectors, the transmission performance can be flexibly adjusted. A commercial dual-polarization IQM could simultaneously generate a chaotic signal and a load message, which provides a high-integration solution. A fast independent component analysis (ICA) algorithm is adopted to compensate for the rotation of state of polarization (RSOP). 60 Gb/s encrypted quadrature phase shift keying (QPSK) signal transmission over 100 km single-mode fiber is realized, and the decrypted bit error rate (BER) performance is below the 7% forward error correction (FEC) threshold (BER = 3.8 × 10-3).