Protein tyrosine phosphatase 1B (PTP1B) is an intractable target for drug discovery due to its conservative and cationic catalytic site. Targeting alternative allosteric sites of PTP1B is a promising strategy to achieve specificity and bioavailability. A hierarchical virtual screening based on a previously identified allosteric site was applied to search for potential PTP1B inhibitors with better pharmacological profiles. Four potent PTP1B inhibitors (H1, H3, H7, and H9) with structures distinct from known inhibitors were identified. Among them, H3 and H9 demonstrated evident selectivity to PTP1B over homologous T-cell protein tyrosine phosphatase (TCPTP) and SHP2. Molecular dynamics simulations and molecular mechanics-generalized Born surface area (MM-GBSA) calculations recognized Phe280, Phe196, Leu192, and Asn193 as key residues responsible for potent allosteric inhibition and excellent PTP selectivity. The results not only expand the structural diversity but also aid the future molecular design of PTP1B allosteric inhibitors.