In this paper, the controlled production of high-quality metal-free diamond nanoparticles is demonstrated. Milling with tempered steel is shown to leave behind iron oxide contamination which is difficult to remove. Milling with SiN alleviates this issue but generates more nondiamond carbon. Thus, the choice of milling materials is critically determined by the acceptable contaminants in the ultimate application. The removal of metal impurities, present in all commercially available nanoparticles, will open new possibilities toward the production of customized diamond nanoparticles, covering the most demanding quantum applications.