Inelastic scattering experiments are key methods for mapping the full dispersion of fundamental excitations of solids in the ground as well as nonequilibrium states. A quantitative analysis of inelastic scattering in terms of phonon excitations requires identifying the role of multiphonon processes. Here, we develop an efficient first-principles methodology for calculating the all-phonon quantum mechanical structure factor of solids. We demonstrate our method by obtaining excellent agreement between measurements and calculations of the diffuse scattering patterns of black phosphorus, showing that multiphonon processes play a substantial role. The present approach constitutes a step towards the interpretation of static and time-resolved electron, x-ray, and neutron inelastic scattering data.