Although 3D positional tracking of single nanoparticles in suspension is now an established technique, the small size of the particles compared to the wavelength of light has meant it is still challenging to optically characterize individual diffusing particles in other ways. Here we introduce Quantitative Optical Anisotropy Imaging (QOAI), an interferometric technique that fills some of this gap by allowing for real-time tracking of orientation as well as spectroscopic characterization of polarizability in nanoparticles at the microsecond timescale. Applying this to gold nanorods, we demonstrate measurement of nanorod orientation with high precision with simultaneous spectroscopic characterization of the rods' longitudinal plasmon resonance. We also show that we can quantify rotational diffusion in individual particles in both the azimuthal and polar directions near a solid wall, as well as detecting binding of particles to that wall. The simple optical configuration of QOAI will make combining it with positional nanoparticle tracking techniques straightforward, and this opens the door to measurements that are not reachable with current techniques, such as detailed characterization of correlations between rotational and translational diffusion in nanoparticles, real-time observation of particle aggregation and assembly, and measurements of fluctuations in the plasmon resonance in metal nanoparticles as they encounter a changing or heterogeneous environment.