Tool wear is one of the main causes of failure during diffraction grating ruling. However, no theoretical model for tool wear analysis has been available to date. A mathematical model is established here to solve for the friction coefficient at the tool contact position for the first time. Based on the ruling principles for diffraction gratings, four parameters comprising the tool cutting edge radius, knife angle, pitch angle, and tool ruling depth, are introduced into the model. The positive pressure and shear stress acting on the tool contact surface element during plastic deformation of the metal film layer are given, and an integral is performed over the area where the tool meets the metal film layer. Equations describing the friction coefficients at different positions on the tip point and the main edge are derived. The friction coefficients at the tip point and main edge positions are then calculated using the model. The cutting edge radius, tool tip angle, and pitch angle are used as variables. The maximum value distribution of the friction coefficients of the anti-wear ruling tool is analyzed, and the principle that parameter selection for the anti-wear ruling tool should meet requirements for a large cutting edge radius, small pitch angle, and large tool tip angle is proposed for the first time. This principle provides the key to solving the technical problem where tool wear occurs easily during ruling of large-area echelle gratings, which has puzzled researchers for many years. Finally, a ruling experiment is performed using a 79 gr/mm echelle grating. Under the large pitch angle condition, the tool jumping phenomenon occurs because of excessive friction force, which results in ruling failure. The numerical analysis results are verified. The research results in this paper can provide a theoretical basis for anti-wear tool design and ruling process optimization.