In this paper, we propose a multi-dimensional multiplexing scheme for space division multiplexing optical transmission systems based on quaternion chaotic encryption. A constellation compression shaping mapping method is designed to replace the traditional 2n mapping scheme, which leads to flexible encoding modulation. In order to achieve orthogonality between data symbols and effectively suppress crosstalk, the spectrally superposed transmission of three-dimensional (3D) constellation data is carried out by code division multiplexing and 4D carrier-less amplitude phase joint modulation techniques based on orthogonal basis. The Chen's chaotic model is utilized to generate the rotation angle of the constellation points, which enables data encryption without changing the transmitted power, enhancing system's security. The feasibility and superiority of the proposed scheme are successfully verified by constructing an experimental platform for a seven-core fiber transmission system.