In this paper, a robust control based on disturbance observer is proposed to improve the tracking accuracy of the Risley prism system (RPS). Applying the flexible thin-wall ring mechanism in the RPS causes a series of tracking and pointing challenges. Disturbances such as friction, shaft deformation, and model perturbation significantly deteriorate the tracking and pointing accuracy of the RPS. Two different observer-based control methods are proposed to guarantee the tracking precision of the RPS. Moreover, the disturbance observation and compensation (DOC) performance of the proposed methods is analyzed and compared. Finally, simulation and experiment results indicate that the proposed control methods, especially the DOC-expanded state observer control mode, obtain the best performance for disturbance rejection in the RPS.