G-protein-coupled receptors (GPCRs) are the largest class of signaling receptors that are most frequently targeted by therapeutic drugs. Allosteric modulators bound to GPCRs at allosteric sites provide the potential for differential selectivity and improved safety compared with traditional orthosteric ligands. The recent breakthroughs in GPCR structural biology have made structures of GPCRs from classes A, B, C, and F complexed with small-molecule allosteric modulators available. Knowledge of the detailed receptor-modulator interactions at the allosteric sites is useful for structure-based GPCR drug design of novel therapeutics. This Perspective comprehensively summarizes the current status of structural complexes between GPCRs and their small-molecule allosteric modulators, particularly the key receptor-modulator interactions at the allosteric sites. Then, the structural diversity of allosteric sites across four GPCR subfamilies is compared. This study is expected to contribute to the design of GPCR allosteric drugs with an improved therapeutic action.