Ultrasmooth surfaces with sub-nanometer roughness and low damage are a great challenge for optical fabrication. Ion beam sputtering (IBS) has obvious advantages on the improvement of surface quality and the removal of surface defects. However, surface defects with different properties and structures display different evolution laws during the IBS process, which affects the roughness change and needs classification studies. In this paper, classification experiments are carried out to study the surface topography evolution of plastic scratches, brittle scratches, and micro-particles during the IBS process. The plastic scratches and micro-particles can be removed, while the brittle scratches can be passivated, so that surface defects can be reduced and surface quality improved. The corresponding evolution mechanisms are discussed in depth, and we show that micro-topography characteristics and material properties are important factors affecting the evolution of surface topography. Through the summary of evolution laws of different surface states, the Gaussian distribution law of surface roughness is established. The evolution regularity and mechanism of surface roughness during the IBS process are expounded upon from the perspective of microscopic morphology, which lays a foundation for ultra-smooth surface manufacturing with low damage.