This paper presents a new dimensional metrological sensing principle for a curved surface based on curved edge diffraction. Spindle error measurement technology utilizes a cylindrical or spherical target artifact attached to the spindle with non-contact sensors, typically a capacitive sensor (CS) or an eddy current sensor, pointed at the artifact. However, these sensors are designed for flat surface measurement. Therefore, measuring a target with a curved surface causes error. This is due to electric fields behaving differently between a flat and curved surface than between two flat surfaces. In this study, a laser is positioned incident to the cylindrical surface of the spindle, and a photodetector collects the total field produced by the diffraction around the target surface. The proposed sensor was compared with a CS within a range of 500 μm. The discrepancy between the proposed sensor and CS was 0.017% of the full range. Its sensing performance showed a resolution of 14 nm and a drift of less than 10 nm for 7 min of operation. This sensor was also used to measure dynamic characteristics of the spindle system (natural frequency 181.8 Hz, damping ratio 0.042) and spindle runout (22.0 μm at 2000 rpm). The combined standard uncertainty was estimated as 85.9 nm under current experiment conditions. It is anticipated that this measurement technique allows for in situ health monitoring of a precision spindle system in an accurate, convenient, and low cost manner.