Platform attitude jitter is inevitable during a satellite flight. The jitter reduces the quality of push-broom optical images, resulting in geometric deformation and image blur. This paper proposes an approach that offers jitter effect compensation and quality improvement for push-broom optical images based on dynamic point spread function (PSF) estimation and iterative image restoration considering the effect of terrain relief and time delay integration (TDI). First, the attitude jitter is precisely detected and estimated by considering the terrain relief. The jitter information is then used to establish the dynamic PSF of each image line considering the changing jitter values and TDI effect. Finally, based on the constructed PSF, the image is improved by iterative restoration using the optimal-window Richardson-Lucy algorithm. The method is validated by both simulation and real data experiments. In the simulation experiment, we apply jitter with different amplitudes and frequencies to generate the degraded images and then restore the image using the proposed restoration method. The results show that the proposed method can effectively restore images affected by jitter. In addition, real data experiments are carried out with multispectral remote sensing images from the ZY-3 satellite. The results show that in addition to the improvement of the radiometric quality, the geometric quality is also significantly improved in both the across-track and the along-track directions. The experimental results validated that the proposed method outperformed other methods without considering the terrain and TDI effect.