Optical coordinate transformation (OCT) has attracted widespread attention in the field of orbital angular momentum (OAM) (de)multiplexing or manipulation, but the performance of OCT would suffer from its distortion. In this paper, we quantitatively analyze the distortion of OCT from the perspective of ray optics and explain its rationality to work under non-normal incident light. For the special case of log-polar coordinate transformation (LPCT), we use a raytracing assisted optimization scheme to improve its distortion, which is related to a Zernike polynomial based phase compensation. After raytracing optimization, the root mean square error (RMSE) of the focused rays is reduced to 1/5 of the original value and the physical optic simulation also shows great improvement. In the experiment, we use three phase masks which are realized by metasurfaces, the measured results show well consistency with the simulation. Results in this paper have great potential to improve the performance of OCT related applications.