Protein tyrosine phosphatases (PTPs) are emerging drug targets for many diseases, including cancer, autoimmunity, and neurological disorders. A high degree of structural similarity between their catalytic domains, however, has hindered the development of selective pharmacological agents. Our previous research uncovered two unfunctionalized terpenoid inhibitors that selectively inhibit PTP1B over T-cell PTP (TCPTP), two PTPs with high sequence conservation. Here, we use molecular modeling, with supporting experimental validation, to study the molecular basis of this unusual selectivity. Molecular dynamics (MD) simulations suggest that PTP1B and TCPTP share a h-bond network that connects the active site to a distal allosteric pocket; this network stabilizes the closed conformation of the catalytically essential WPD loop, which it links to the L-11 loop and neighboring α3 and α7 helices on the other side of the catalytic domain. Terpenoid binding to either of two proximal C-terminal sites─an α site and a β site─can disrupt the allosteric network; however, binding to the α site forms a stable complex only in PTP1B. In TCPTP, two charged residues disfavor binding at the α site in favor of binding at the β site, which is conserved between the two proteins. Our findings thus indicate that minor amino acid differences at the poorly conserved α site enable selective binding, a property that might be enhanced with chemical elaboration, and illustrate more broadly how minor differences in the conservation of neighboring─yet functionally similar─allosteric sites can affect the selectivity of inhibitory scaffolds (e.g., fragments).