In the field of automatic optical inspection, it is imperative to measure the defects on spherical optical surfaces. So a novel spherical surface defect evaluation system is established in this paper to evaluate defects on optical spheres. In order to ensure the microscopic scattering dark-field imaging of optical spheres with different surface shape and radius of curvature, illumination with variable aperture angle is employed. In addition, the scanning path of subapertures along the parallels and meridians is planned to detect the large optical spheres. Since analysis shows that the spherical defect information could be lost in the optical imaging, the three-dimensional correction based on a pin-hole model is proposed to recover the actual spherical defects from the captured two-dimensional images. Given the difficulty of subaperture stitching and defect feature extraction in three-dimensional (3D) space after the correction, the 3D subapertures are transformed into a plane to be spliced through geometric projection. Then, methods of the surface integral and calibration are applied to quantitatively evaluate the spherical defects. Furthermore, the 3D panorama of defect distribution on the spherical optical components can be displayed through the inverse projective reconstruction. Finally, the evaluation results are compared with the OLYMPUS microscope, testifying to the micrometer resolution, and the detection error is less than 5%.