ERAP1 and ERAP2 are endoplasmic reticulum zinc-binding aminopeptidases that play crucial roles in processing peptides for loading onto class I major histocompatibility complex proteins. These enzymes are therapeutic targets in cancer and autoimmune disorders. The discovery of inhibitors specific to ERAP1 or ERAP2 has been challenging due to the similarity in their active site residues and domain architectures. Here, we identify 4-methoxy-3-{[2-piperidin-1-yl-4-(trifluoromethyl) phenyl] sulfamoyl} benzoic acid (compound 61) as a novel inhibitor of ERAP2 and determine the crystal structure of ERAP2 bound to compound 61. Compound 61 binds near the catalytic center of ERAP2, at a distinct site from previously known peptidomimetic inhibitors, and inhibits by an uncompetitive mechanism. Surprisingly, for ERAP1, compound 61 was found to activate model substrate hydrolysis, similarly to the previously characterized 5-trifluoromethyl regioisomer of compound 61, known as compound 3. We characterized the specificity determinants of ERAP1 and ERAP2 that control the binding of compounds 3 and 61. At the active site of ERAP1, Lys380 in the S1' pocket is a key determinant for the binding of both compounds 3 and 61. At the allosteric site, ERAP1 binds either compound, leading to the activation of model substrate hydrolysis. Although ERAP2 substrate hydrolysis is not activated by either compound, the mutation of His904 to alanine reveals a cryptic allosteric site that allows for the activation by compound 3. Thus, we have identified selectivity determinants in the active and allosteric sites of ERAP2 that govern the binding of two similar compounds, which potentially could be exploited to develop more potent and specific inhibitors.