To make the surface defects evaluation system (SDES) of fine flat optics more effective and reliable, the point-like defects on the surface are divided into two categories: digs and dust particles. Since only the digs are the real damages that should be sent for further investigation, the false signals associated with dust particles should be distinguished and removed. Dark-field scattering microscopy and pattern recognition methodology are combined to classify digs and dust particles. The SDES is employed for dark-field image acquisition of optical samples. Gray scale, texture, and morphology analyses are then conducted on each image to extract raw feature data, which are compressed with the principal component analysis. Based on the compressed feature data, the support vector machine is used to construct the classification model. The success discrimination rates are 96.56% for the training set and 93.90% for the prediction set. The classification results are presented to show the potential of this method to be used for practical digs and dust particles discrimination on the actual optical samples.