We report on the use of a thin diffuser placed in the close vicinity of a camera sensor as a simple and effective way to superlocalize plasmonic nanoparticles in 3D. This method is based on holographic reconstruction via quantitative phase and intensity measurements of a light field after its interaction with nanoparticles. We experimentally demonstrate that this thin diffuser can be used as a simple add-on to a standard bright-field microscope to allow the localization of 100 nm gold nanoparticles at video rate with nanometer precision (1.3 nm laterally and 6.3 nm longitudinally). We exemplify the approach by revealing the dynamic Brownian trajectory of a gold nanoparticle trapped in various pockets within an agarose gel. The proposed method provides a simple but highly performant way to track nanoparticles in 3D.