Optical coherence tomography (OCT) can map the stiffness of biological tissue by imaging mechanical perturbations (shear waves) propagating in the tissue. Most shear wave elastography (SWE) techniques rely on active shear sources to generate controlled displacements that are tracked at ultrafast imaging rates. Here, we propose a noise-correlation approach to retrieve stiffness information from the imaging of diffuse displacement fields using low-frame rate spectral-domain OCT. We demonstrated the method on tissue-mimicking phantoms and validated the results by comparison with classic ultrafast SWE. Then we investigated the in vivo feasibility on the eye of an anesthetized rat by applying noise correlation to naturally occurring displacements. The results suggest a great potential for passive elastography based on the detection of natural pulsatile motions using conventional spectral-domain OCT systems. This would facilitate the transfer of OCT-elastography to clinical practice, in particular, in ophthalmology or dermatology.