In this article, a distributed active disturbance rejection formation tracking control strategy is developed for a quadrotor unmanned aerial vehicle (UAV) swarm system with switching communication topologies. The proposed control strategy consists of two parts: 1) attitude-loop control and 2) position-loop control. First, a distributed cascade active disturbance rejection control (CADRC) method is designed for the attitude subsystem. With this attitude control method, the stability of the attitude subsystem can be achieved even in the presence of unknown time-varying disturbances. Second, a distributed formation tracking control method is designed for the position subsystem. This position control method ensures that the quadrotor UAV swarm maintains dynamic formation flying and accurately tracks a predetermined trajectory. Through stability analysis, it can be proved that the proposed control strategy can ensure the stability of the whole swarm system. Finally, the proposed control strategy is applied to a quadrotor UAV swarm system to verify its effectiveness and the ability to suppress the influence of unknown time-varying disturbances.