In recent years, with the development of information networks, higher requirements for transmission capacity have been recommended. Yet, at the same time, the capacity of single-mode fiber is rapidly approaching the theoretical limit. The multidimensional multiplexing technique is an effective way to solve this problem. Since the high differential mode delay (DMD) of transmission fiber increases the complexity of demultiplexing in equalization algorithms, we use an intelligent design method to optimize the trench-assisted gradient refractive index structure in this paper. The maximum DMD of the optimized optical fiber structure is 19.6 ps/km. A least mean squares-feedforward neural network constant modulus algorithm (LMS-FNNCMA) is also designed by using the theory of the least mean squares (LMS), constant modulus algorithm (CMA), and the multiple input multiple output (MIMO) neural networks. In order to verify the accuracy of the algorithm, a polarization division multiplexing-wavelength division multiplexing-mode division multiplexing (PDM-WDM-MDM) optical transmission system is constructed through simulation. The algorithm successfully realizes the de-crosstalk over a transmission distance of 1200 km at a rate of 1.2 Tbps under simulation conditions.