Metal nanoparticles play a key role in sensing and imaging. Here we demonstrate the detection of metal particles down to 5 nm in size with a signal-to-noise ratio of ∼7 using interferometric cross-polarization microscopy at ultralow excitation powers (∼1 μW) compatible with single molecule detection. The method is background-free and induces no heating as it operates far from plasmonic resonance. The combination of unlimited observation time and protein-sized metal nanoparticles has great potential for biophysical applications.