We take a data-driven approach to deducing the local volume changes accompanying early development of the fetal human brain. Our approach uses fetal brain atlas MRI data for the geometric changes in representative cases. Using a nonlinear continuum mechanics model of morphoelastic growth, we invert the deformation obtained from MRI registration to arrive at a field for the growth deformation gradient tensor. Our field inversion uses a combination of direct and adjoint methods for computing gradients of the objective function while constraining the optimization by the physics of morphoelastic growth. We thus infer a growth deformation gradient field that obeys the laws of morphoelastic growth. The errors between the MRI data and the forward displacement solution driven by the inverted growth deformation gradient field are found to be smaller than the reference displacement by well over an order of magnitude, and can be driven even lower. The results thus reproduce the three-dimensional growth during the early development of the fetal brain with controllable error. Our findings confirm that early growth is dominated by in plane cortical expansion rather than thickness increase.