Diffusing wave spectroscopy (DWS) is a group of techniques used to measure the dynamics of a scattering medium in a non-invasive manner. DWS methods rely on detecting the speckle light field from the moving scattering medium and measuring the speckle decorrelation time to quantify the scattering medium's dynamics. For DWS, the signal-to-noise (SNR) is determined by the ratio between measured decorrelation time to the standard error of the measurement. This SNR is often low in certain applications because of high noise variances and low signal intensity, especially in biological applications with restricted exposure and emission levels. To address this photon-limited signal-to-noise ratio problem, we investigated, theoretically and experimentally, the SNR of an interferometric speckle visibility spectroscopy (iSVS) compared to more traditional DWS methods. We found that iSVS can provide excellent SNR performance through its ability to overcome camera noise. We also proved an iSVS system has more relaxed constraints on the reference beam properties. For an iSVS system to function properly, we only require the reference beam to exhibit local temporal stability, while incident angle, reference phase and intensity uniformity do not need to be constrained. This flexibility can potentially enable more unconventional iSVS implementation schemes.