Scatter can account for large errors in cone-beam CT (CBCT) due to its wide field of view, and its complicated nature makes its compensation difficult. Iterative polyenergetic reconstruction algorithms offer the potential to provide quantitative imaging in CT, but they are usually incompatible with scatter contaminated measurements. In this work, we introduce a polyenergetic convolutional scatter model that is directly fused into the reconstruction process, and exploits information readily available at each iteration for a fraction of additional computational cost. We evaluate this method with numerical and real CBCT measurements, and show significantly enhanced electron density estimation and artifact mitigation over pre-calculated fast adaptive scatter kernel superposition (fASKS). We demonstrate our approach has two levels of benefit: reducing the bias introduced by estimating scatter prior to reconstruction; and adapting to the spectral and spatial properties of the specimen.