A novel nonparaxial scalar theory is presented to calculate the angular scattering that is due to interface roughnesses or bulk inhomogeneities in a high-quality optical coating. Based on the empirically modified Beckmann-Kirchhoff surface scatter model, this theory in surface scattering and bulk scattering predicts similar formulas for the angular scattered intensity, and at the same time provides new understanding and insight into multilayer scattering phenomena. It is worth noting that the derived expressions are in the same form as those given by the typical vector methods. Based on comparisons of the surface and bulk models with the corresponding typical models for several multilayer designs, the novel theory is demonstrated to be valid for multilayer coatings even with large incident and scattering angles.