A brush seal has the advantages of adapting to different vibration conditions and increasing the stability of the nonlinear rotor system. In this research, the stability and bifurcations of complex vibrations in a brush-seal rotor system are studied. An analytical seal force model is obtained through the beam theory and mutual coupling dynamics of the bristles and the rotor. The interaction between the bristles and the rotor is clearly depicted by a geometric map. Periodic and chaotic vibrations as well as the corresponding amplitude-frequency characteristics are first predicted by a numerical bifurcation diagram and 3D waterfalls. Discrete dynamic eigenvalue analysis is adopted for a detailed investigation of the stability and bifurcations of nonlinear vibrations. Jumping, quasi-periodic, and half-frequency vibrations are warned during the speeding up and down process. Four separate nonlinear vibration evolving routes are discovered. Two period-doubling bifurcation trees evolving to chaos are illustrated for the observation of global and independent periodic vibrations. Nonlinear vibration illustrations are presented through displacement orbits as well as harmonic amplitudes and phases. Chaotic vibration and unstable semi-analytical vibration solutions are compared. The obtained results and analysis methods provide new perspectives on nonlinear vibrations in the brush-seal rotor system.