We demonstrate a universal correlation between the spectral linewidth and position of the excitonic transition in the spectral jitter observed from single elongated colloidal quantum dots. Breaking the symmetry of electron and hole confinement as well as of the spatial directions for surface charge diffusion enables us to microscopically track meandering surface charges, providing a novel probe of the particle's nanoenvironment. Spectral diffusion exhibits only a weak temperature dependence, which allows us to uncover the single particle homogeneous linewidth of 50 meV at room temperature.