The advent of monochromated electron energy-loss spectroscopy has enabled atomic-resolution vibrational spectroscopy, which triggered interest in spatially localized or quasi-localized vibrational modes in materials. Here we report the discovery of phonon vortices at heavy impurities in two-dimensional materials. We use density-functional-theory calculations for two configurations of Si impurities in graphene, Si-C3 and Si-C4, to examine atom-projected phonon densities of states and display the atomic-displacement patterns for select modes that are dominated by impurity displacements. The vortices are driven by large displacements of the impurities, and reflect local symmetries. Similar vortices are found at phosphorus impurities in hexagonal boron nitride, suggesting that they may be a feature of heavy impurities in crystalline materials. Phonon vortices at defects are expected to play a role in thermal conductivity and other properties.