Detection accuracy is an important performance indicator of ground-based telescopes and is affected mainly by pointing error, geometric distortion of the optical system, and parameter errors caused by machining error and installation error. To improve detection accuracy, a modified algorithm based on a simulated annealing algorithm is proposed in this paper; this algorithm is able to correct pointing, derive a geometric distortion solution, and re-estimate some parameters of telescopes simultaneously. The efficiency of the proposed method is verified by using the observation data of the telescope, whose aperture is 600 mm under two distortion models (the physical model and polynomial fitting model). The results show that the method presented in this paper can effectively solve the problem of nonconvergence of the distortion solution with a pointing error. The final angle error under the polynomial fitting model is 1.07″, and the pixel error is 0.06 pixels; the errors under the physical model are 1.08″ and 0.07 pixels. The correction effect under the two distortion models is basically the same, but the averaged operation speed based on the physical model is 19.45% faster.