A rotation angle measurement method based on self-mixing interference (SMI) of a fiber laser is proposed. The rotation angle can be calculated indirectly by the displacement measured by SMI. In the experiment, a linear cavity fiber laser with simple structure and high flexibility is used as the optical source for measuring the deflection angle. To improve the measurement accuracy, the SMI signal is filtered by the variational mode decomposition (VMD) algorithm. The filtered SMI signal is normalized by Hilbert transform. The even-power algorithm is used to subdivide the interference fringes, so as to improve the measurement resolution. The experimental result shows that the measurement error of angular shift is less than 1% in the range of 10°.