To making the high resolution grating, a numerical calculation was used to analyze the effect of recording parameters on groove density, focal curve and imaging performance of the grating and their compensation. Based on Fermat' s principle, light path function and aberration, the effect on imaging performance of the grating was analyzed. In the case of fixed using parameters, the error of the recording angle has a greater influence on imaging performance, therefore the gain of the weight of recording angle can improve the accuracy of the recording angle values in the optimization; recording distance has little influence on imaging performance; the relative errors of recording parameters cause the change of imaging performance of the grating; the results indicate that recording parameter errors can be compensated by adjusting its corresponding parameter. The study can give theoretical guidance to the fabrication for high resolution varied-line-space plane holographic grating in on-line spectral diagnostic and reduce the alignment difficulty by analyze the main error effect the imaging performance and propose the compensation method.