Swarming behavior in systems of self-propelled particles, whether biological or artificial, has received increased attention in recent years. Here, we show that even a small number of particles with anomalous behavior can change dramatically collective dynamics of the swarming system and can impose unusual behavior and transitions between dynamic states. Our results pave the way to practical approaches and concepts of multiagent dynamics in groups of flocking animals: birds, insects, and fish, i.e., active and living soft matter.