In this paper, a security enhanced physical layer encryption scheme is proposed for coherent optical polarization division multiplexing (PDM) systems. The concept of a digital optical polarization scrambler (DOPS) is introduced to apply high speed rotation of state of polarization (RSOP) to the transmitted signal, which enables encryption based on polarization perturbations and offers superior flexibility in polarization management. By utilizing different combinations of digital polarization device matrices and adjusting their key parameters, four encryption modes are designed. The proposed encryption scheme is successfully implemented in a PDM-QPSK system at the data rate of 32 Gbps. Experimental results demonstrate that authorized users can successfully decrypt the received signal, while the eavesdroppers cannot derive useful information with a bit error rate (BER) at approximately 0.5. To enhance system security, a 5-D chaotic system is introduced with superior properties of high sensitivity to initial values and improved uniform distribution, which guarantees the large entropy and further the system's security. This scheme can effectively prevent against brute attacks with the expanded key space of 1060.