Here, we present a concept based on the realization that a complex medium can be used as a simple interferometer. Changes in the wavefront of an incident coherent beam can be retrieved by analyzing changes in speckle patterns when the beam passes through a light diffuser. We demonstrate that the spatial intensity correlations of the speckle patterns are independent of the light diffusers, and are solely determined by the phase changes of an incident beam. With numerical simulations using the random matrix theory, and an experimental pressure-driven wavefront-deforming setup using a microfluidic channel, we theoretically and experimentally confirm the universal sensitivity of speckle intensity correlations, which is attributed to the conservation of optical field correlation despite multiple light scattering. This work demonstrates that a light diffuser works as a simple interferometer, and presents opportunities to retrieve phase information of optical fields with a compact scattering layer in various applications in metrology, analytical chemistry, and biomedicine.