The understanding and manipulation of anisotropic Gilbert damping is crucial for both fundamental research and versatile engineering and optimization. Although several works on anisotropic damping have been reported, no direct relationship between the band structure and anisotropic damping was established. Here, we observed an anisotropic damping in Fe/GeTe manipulated by the symmetric band structures of GeTe via angle-resolved photoemission spectroscopy. Moreover, the anisotropic damping can be modified by the symmetry of band structures. Our Letter provides insightful understandings of the anisotropic Gilbert damping in ferromagnets interfaced with Rashba semiconductors and suggests the possibility of manipulating the Gilbert damping by band engineering.