Phase-sensitive optical coherence tomography (OCT) is used to measure motion in a range of techniques, such as Doppler OCT and optical coherence elastography (OCE). In phase-sensitive OCT, motion is typically estimated using a model of the OCT signal derived from a single reflector. However, this approach is not representative of turbid samples, such as tissue, which exhibit speckle. In this study, for the first time, we demonstrate, through theory and experiment that speckle significantly lowers the accuracy of phase-sensitive OCT in a manner not accounted for by the OCT signal-to-noise ratio (SNR). We describe how the inaccuracy in speckle reduces phase difference sensitivity and introduce a new metric, speckle brightness, to quantify the amount of constructive interference at a given location in an OCT image. Experimental measurements show an almost three-fold degradation in sensitivity between regions of high and low speckle brightness at a constant OCT SNR. Finally, we apply these new results in compression OCE to demonstrate a ten-fold improvement in strain sensitivity, and a five-fold improvement in contrast-to-noise by incorporating independent speckle realizations. Our results show that speckle introduces a limit to the accuracy of phase-sensitive OCT and that speckle brightness should be considered to avoid erroneous interpretation of experimental data.