There is growing interest in using the exquisite phase sensitivity of optical coherence tomography (OCT) to measure the vibratory response in organ systems such as the middle and inner ear. Using frequency domain analysis, it is possible to achieve picometer sensitivity to vibration over a wide frequency band. Here we explore the limits of the frequency domain vibratory sensitivity due to additive noise and consider the implication of phase noise statistics on the estimation of vibratory amplitude and phase. Noise statistics are derived in both the Rayleigh (s/n = 0) and Normal distribution (s/n > 3) limits. These theoretical findings are explored using simulation and verified with experiments using a swept-laser system and a piezo electric element. A metric for sensitivity is proposed based on the 98% confidence interval for the Rayleigh distribution.