We present experimental results of chaotic mixing of Newtonian fluids and yield-stress fluids using a rod-stirring protocol with a rotating vessel. We show how the mixing of yield-stress fluids by chaotic advection is reduced compared to the mixing of Newtonian fluids and explain our results, bringing to light the relevant mechanisms: the presence of fluid that only flows intermittently, a phenomenon enhanced by the yield stress, and the importance of the peripheral region. This finding is confirmed via numerical simulations. Anomalously slow mixing is observed when the synchronization of different stirring elements leads to the repetition of slow stretching for the same fluid particles.