In this work we propose a mechanism to explain the enhancement of the magnetic-field-induced yield stress when nonmagnetic particles are added to magnetic particulate suspensions, i.e., two-component suspensions. Our main hypothesis is that the nonmagnetic particles collide with the field-induced magnetic aggregates under shear flow. Consequently, supplementary fluctuations of the orientations of the magnetic aggregates occur, resulting in an effective rotary diffusion process, which increases the dynamic yield stress of the suspension. Furthermore, the collision rate and the rotary diffusivity of the aggregates should increase with the concentration of nonmagnetic particles. Rheological measurements in plate-plate and cylindrical Couette geometries confirm the increase of the yield stress with the volume fraction of nonmagnetic particles. In addition, such an effect appears to be more important in Couette geometry, for which orientation fluctuations of the magnetic aggregates play a more significant role. Finally, a theoretical model based on this rotary diffusion mechanism is developed, providing a quantitative explanation of the experimentally observed trends.