A model of closed-loop retinal motion tracking in an adaptive optics scanning light ophthalmoscope (AOSLO) was built, and the tracking performance was optimized by minimizing the root-mean-square of residual motion. We started with an evaluation of the fidelity of the retinal motion measurement, and then analyzed the transfer function of the system and power spectral density of retinal motion from human eyes, to achieve optimal control gain and sampling frequency. The performance was further enhanced by incorporating retinal motion prediction during the period in which the slow scanner was retracing. After optimization, residual image motion performance was improved by 33% with a nearly 50% reduction in computational cost in comparison to our previous setup, reaching a 3 dB bandwidth of 15-17 Hz, which is close to the frame rate (∼21 fps) of this AOSLO system.