It has been recently shown [E. Lerner, G. Düring, and E. Bouchbinder, Phys. Rev. Lett. 117, 035501 (2016)PRLTAO0031-900710.1103/PhysRevLett.117.035501] that the nonphononic vibrational modes of structural glasses at low frequencies ω are quasilocalized and follow a universal density of states D(ω)∼ω^{4}. Here we show that the gapless nature of the observed density of states depends on the existence of internal stresses that generically emerge in glasses due to frustration, thus elucidating a basic element underlying this universal behavior. Similarly to jammed particulate packings, low-frequency modes in structural glasses emerge from a balance between a local elasticity term and an internal stress term in the dynamical matrix, where the difference between them is orders of magnitude smaller than their typical magnitude. By artificially reducing the magnitude of internal stresses in a computer glass former in three dimensions, we show that a gap is formed in the density of states below which no vibrational modes exist, thus demonstrating the crucial importance of internal stresses. Finally, we show that while better annealing the glass upon cooling from the liquid state significantly reduces its internal stresses, the self-organizational processes during cooling render the gapless D(ω)∼ω^{4} density of state unaffected.