SHP2 is a protein tyrosine phosphatase (PTP) involved in multiple signaling pathways. Mutations of SHP2 can result in Noonan syndrome or pediatric malignancies. Inhibition of wild-type SHP2 represents a novel strategy against several cancers. SHP2 is activated by binding of a phosphopeptide to the N-SH2 domain of SHP2, thereby favoring dissociation of the N-SH2 domain and exposing the active site on the PTP domain. The conformational transitions controlling ligand affinity and PTP dissociation remain poorly understood. Using molecular simulations, we revealed an allosteric interaction restraining the N-SH2 domain into a SHP2-activating and a stabilizing state. Only ligands selecting for the activating N-SH2 conformation, depending on ligand sequence and binding mode, are effective activators. We validate the model of SHP2 activation by rationalizing modified basal activity and responsiveness to ligand stimulation of several N-SH2 variants. This study provides mechanistic insight into SHP2 activation and may open routes for SHP2 regulation.