Diffusion Magnetic Resonance Imaging provides images of unquestionable diagnostic value. It is commonly used in the assessment of stroke and in white matter fiber tracking, among other applications. The diffusion coefficient has been shown to depend on cell concentration, membrane permeability, and cell orientation in the case of white matter or muscle fiber tracking; yet a clear relation between diffusion measurements and known physiological parameters is not established. The aim of this paper is to review hypotheses and actual knowledge on diffusion signal origin to provide assistance in the interpretation of diffusion MR images. Focus will be set on brain images, as most common applications of diffusion MRI are found in neuroradiology. Diffusion signal does not come from two intra- or extracellular compartments, as was first assumed. Restriction of water displacement due to membranes, hindrance in the extracellular space, and tissue heterogeneity are important factors. Unanswered questions remain on how to deal with tissue heterogeneity, and how to retrieve parameters less troublesome to work with from biological and clinical points of view. Diffusion quantification should be done with care, as many variables can lead to variation in measurements.