Direct measurement of tissue microstructure with diffusion MRI offers a new class of biomarkers, such as axon diameters, that give more specific information about tissue than measures derived from diffusion tensors. The existing techniques of this kind assume a single axon orientation in the tissue model, which may be a reasonable approximation only for the most coherent brain white matter, such as the corpus callosum. For most other areas, orientation dispersion is not negligible and, if unaccounted for, leads to overestimation of the axon diameters, prohibiting their accurate mapping over the whole brain. Here we propose a new model that captures the effect of orientation dispersion explicitly, An efficient numerical scheme is developed to enable the axon diameter estimation by fitting the proposed model. Synthetic data experiments demonstrate that the new model provides an axon diameter index that is robust to the presence of orientation dispersion. Results on in vivo human data show reduced axon diameter index and better agreement with histology compared to previous methods suggesting improvements in the axon diameter estimate.