Strong acid, alkali, or active chemicals are generally employed in chemical mechanical polishing (CMP) slurry for the ultra-precision machining of semiconductor substrates, obstructing further improvement in machining accuracy and reduction in damaged layers. Understanding the material removal behavior of abrasives, which combines mechanical and chemical actions during nanoscale abrasive polishing, poses a significant challenge. In this study, the interfacial interaction mechanisms between abrasives and substrates during nanoscale polishing with only deionized water used as coolant were analyzed by utilizing concerned experiments and molecular dynamics (MD) simulations. The results showed that ultra-smooth SiC and sapphire substrate surfaces only with damage layer thicknesses of 2.06 nm and 0.76 nm were obtained by employing diamond abrasive and alumina abrasive, respectively. Tribochemical reactions could occur on substrate surfaces under the repeated interfacial fiction of fine abrasives without any chemical reagents during nanoscale polishing. The tribochemical reactions mainly depended on the chemical compatibility between the abrasives and substrates when the material removal size of abrasives was limited to nanoscale, not abrasive hardness. Furthermore, a new method of abrasive selection is proposed and widely applied in the ultra-precision polishing of semiconductor substrates, which could not only improve removal efficiency but also ensure good surface quality.