Current clinical MRI techniques in rectal cancer have limited ability to examine cancer stroma. The differentiation of tumour from desmoplasia or fibrous tissue remains a challenge. Standard MRI cannot differentiate stage T1 from T2 (invasion of muscularis propria) tumours. Diffusion tensor imaging (DTI) can probe tissue structure and organisation (anisotropy). The purpose of this study was to examine DTI-MRI derived imaging markers of rectal cancer stromal heterogeneity and tumour extent ex vivo. DTI-MRI at ultra-high magnetic field (11.7 tesla) was used to examine the stromal microstructure of malignant and normal rectal tissue ex vivo, and the findings were correlated with histopathology. Images obtained from DTI-MRI (A0, apparent diffusion coefficient and fractional anisotropy (FA)) were used to probe rectal cancer stromal heterogeneity. FA provided the best discrimination between cancer and desmoplasia, fibrous tissue and muscularis propria. Cancer had relatively isotropic diffusion (mean FA 0.14), whereas desmoplasia (FA 0.31) and fibrous tissue (FA 0.34) had anisotropic diffusion with significantly higher FA than cancer (p < 0.001). Tumour was distinguished from muscularis propria (FA 0.61) which was highly anisotropic with higher FA than cancer (p < 0.001). This study showed that DTI-MRI can assist in more accurately defining tumour extent in rectal cancer.