AMPylation, the post-translational modification with adenosine monophosphate (AMP), is catalyzed by effector proteins from a variety of pathogens. Legionella pneumophila is thus far the only known pathogen that, in addition to encoding an AMPylase (SidM/DrrA), also encodes a deAMPylase, called SidD, that reverses SidM-mediated AMPylation of the vesicle transport GTPase Rab1. DeAMPylation is catalyzed by the N-terminal phosphatase-like domain of SidD. Here, we determined the crystal structure of full length SidD including the uncharacterized C-terminal domain (CTD). A flexible loop rich in aromatic residues within the CTD was required to target SidD to model membranes in vitro and to the Golgi apparatus within mammalian cells. Deletion of the loop (Δloop) or substitution of its aromatic phenylalanine residues rendered SidD cytosolic, showing that the hydrophobic loop is the primary membrane-targeting determinant of SidD. Notably, deletion of the two terminal alpha helices resulted in a CTD variant incapable of discriminating between membranes of different composition. Moreover, a L. pneumophila strain producing SidDΔloop phenocopied a L. pneumophila ΔsidD strain during growth in mouse macrophages and displayed prolonged co-localization of AMPylated Rab1 with LCVs, thus revealing that membrane targeting of SidD via its CTD is a critical prerequisite for its ability to catalyze Rab1 deAMPylation during L. pneumophila infection.