In this paper, a novel three-dimensional selective probabilistic shaping (3D-SPS) and chaos-based multi-stage encryption scheme is proposed for physical layer security enhancement and transmission performance improvement in orthogonal frequency division multiplexing-based passive optical network (OFDM-PON). On the basis of inherent randomness of symbol sub-sequences with low granularity, the SPS algorithm is performed on the employed cubic constellation within each sub-sequence. Consequently, the probability distribution of inner points significantly increases after the constellation region exchange according to various rules. The generated compressed shaping information (CSI) is encrypted and used as the synchronization head for transmission. Furthermore, 3D scrambling is performed while maintaining the shaping effect. The encrypted signals of 35.3 Gb/s are successfully transmitted over a 25-km standard single-mode fiber (SSMF) and a back-to-back (BTB) system. The results show that by selecting the appropriate system parameter, the proposed scheme can provide about 2.4 dB modulation gain on the received optical power at a bit error rate (BER) of 10‒3 compared with a conventional quadrature amplitude modulation (QAM) signal under the same bit rate, and 0.9 dB shaping gain is brought due to the SPS. The encryption method possesses a relatively low computational complexity and sufficient key space of 10120 is introduced to resist exhaustive attack.