Glutaminyl cyclase (QC) has been implicated in the formation of toxic amyloid plaques by generating the N-terminal pyroglutamate of β-amyloid peptides (pGlu-Aβ) and thus may participate in the pathogenesis of Alzheimer's disease (AD). We designed a library of glutamyl cyclase (QC) inhibitors based on the proposed binding mode of the preferred substrate, Aβ3E-42. An in vitro structure-activity relationship study identified several excellent QC inhibitors demonstrating 5- to 40-fold increases in potency compared to a known QC inhibitor. When tested in mouse models of AD, compound 212 significantly reduced the brain concentrations of pyroform Aβ and total Aβ and restored cognitive functions. This potent Aβ-lowering effect was achieved by incorporating an additional binding region into our previously established pharmacophoric model, resulting in strong interactions with the carboxylate group of Glu327 in the QC binding site. Our study offers useful insights in designing novel QC inhibitors as a potential treatment option for AD.