The diverse Regulator of G protein Signaling (RGS) family sets the timing of G protein signaling. To understand how the structure of RGS proteins determines their common ability to inactivate G proteins and their selective G protein recognition, we combined structure-based energy calculations with biochemical measurements of RGS activity. We found a previously unidentified group of variable 'Modulatory' residues that reside at the periphery of the RGS domain-G protein interface and fine-tune G protein recognition. Mutations of Modulatory residues in high-activity RGS proteins impaired RGS function, whereas redesign of low-activity RGS proteins in critical Modulatory positions yielded complete gain of function. Therefore, RGS proteins combine a conserved core interface with peripheral Modulatory residues to selectively optimize G protein recognition and inactivation. Finally, we show that our approach can be extended to analyze interaction specificity across other large protein families.