A microscopic activated barrier hopping theory of the viscoelasticity of colloidal glasses and gels has been generalized to treat the nonlinear rheological behavior of particle-polymer suspensions. The quiescent cage constraints and depletion bond strength are quantified using the polymer reference interaction site model theory of structure. External deformation (strain or stress) distorts the confining nonequilibrium free energy and reduces the barrier. The theory is specialized to study a limiting mechanical description of yielding and modulus softening in the absence of thermally induced barrier hopping. The yield stress and strain show a rich functional dependence on colloid volume fraction, polymer concentration, and polymer-colloid size asymmetry ratio. The yield stress collapses onto a master curve as a function of the polymer concentration scaled by its ideal mode-coupling gel boundary value, and sufficiently deep in the gel is of an effective power-law form with a universal exponent. A similar functional and scaling dependence of the yield stress on the volume fraction is found, but the apparent power-law exponent is nonuniversal and linearly correlated with the critical gel volume fraction. Stronger gels are generally, but not always, predicted to be more brittle in the strain mode of deformation. The theoretical calculations appear to be in accord with a broad range of observations.