One overarching challenge of clinical magnetic resonance imaging (MRI) is to quantify tissue structure at the cellular scale of micrometers, based on an MRI acquisition with a millimeter resolution. Diffusion MRI (dMRI) provides the strongest sensitivity to the cellular structure. However, interpreting dMRI measurements has remained a highly ill-posed inverse problem. Here we propose a framework that resolves the above challenge for human white matter fibers, by unifying intra-voxel mesoscopic modeling with global fiber tractography. Our algorithm is based on a Simulated Annealing approach which simultaneously optimizes diffusion parameters and fiber locations. Each fiber carries its by their individual set of diffusion parameters which allows to link them structural relationships.