Ensemble particle sizing has traditionally relied on inversion of extinction measurements for the characterization of the particle size distribution (PSD) in particulate media. However, particulate media induce complex phase changes that contain valuable information about their structure. Here, we propose the use of coherent detection to derive particle size distributions in inhomogeneous samples from light scattering. This is achieved by exploiting THz waves, which allow for both extinction and refractive index information to be directly retrieved. A modified version of the iterative Twomey method is presented in order to take into account this information. Additionally, by using a forward model based on the Waterman-Truell formula for the complex refractive index, samples with absorption in both the matrix medium and the particulate phase can be measured. The inversion needs neither a priori assumptions nor constraints regarding the PSD shape. Numerical simulations show that this full-field approach reduces the error of the inversion process potentially up to 65% compared with inversion using only extinction data. Experimental validation of the technique is provided by measuring calibrated spherical glass particles inside a PTFE matrix and retrieving the PSD in the case of monodisperse and polydisperse samples showing an enhancement of up to 32% in comparison to inversion from extinction data.