In this article, a fractional-order sliding mode control (FOSMC) scheme is proposed for mitigating harmonic distortions in the power system, whereby a self-constructing recurrent fuzzy neural network (SCRFNN) is used to weaken the effect of compound nonlinearity caused by unknown uncertainties and environmental fluctuations. The fractional-order sliding mode controller (SMC) is constructed to maintain the control system to be asymptotically stable and a fractional-order calculus is introduced into an SMC to soften the sliding manifold design and realize chattering reduction. Considering parameter variations existing in the power system model, SCRFNN is adopted to approximate the unknown dynamics, which is able to dynamically update network structure by optimizing the fuzzy division, and a feedback connection is incorporated into the feedforward neural network, which is regarded as a storage unit to enhance the capability of coping with temporal problem. The control scheme combining the FOSMC with the SCRFNN can make the tracking error and its time derivative converge to zero. Experimental studies demonstrate the validity of the designed scheme, and comprehensive comparisons illustrate its superiority in harmonic suppression and high robustness.