When encountering a hostile particle, the avoidance behaviors of the vortex state of self-propelled particles exhibit phase transition phenomena such that the vortex state can change into a crystal state. Based on the self-propelled particle model and a molecular dynamics simulation, the dynamic response of the vortex swarm induced by a hostile particle (predator or obstacle) is studied. Three parameters are defined to characterize the collective escaping behaviors, including the order parameter, the flock size, and the roundness parameter. If a predator moves slower with a larger risk radius, the vortex swarm cannot return to its original vortex state, but rather transforms into a crystal state. The critical phase transition radius, the maximum risk radius of a predator with which the transition from a vortex to crystal state cannot take place, is also examined by considering the influence of the model parameters. To some degree, the critical radius reflects the stability and robustness of the vortex swarm.