Modification of chromatin and related transcription factors by histone deacetylases (HDACs) is one of the major strategies for controlling gene expression in eukaryotes. The HDAC domains of class IIa HDACs repress the respective target genes by interacting with the C-terminal region of the silencing mediator for retinoid and thyroid receptor (SMRT) repression domain 3 (SRD3c). However, latent catalytic activity suggests that their roles as deacetylases in gene regulation are unclear. Here, we found that two conserved GSI-containing motifs of SRD3c are critical for HDAC4 binding. Two SMRT peptides including these motifs commonly form a β-hairpin structure in the cleft and block the catalytic entry site of HDAC4. They interact mainly with class IIa HDAC-specific residues of HDAC4 in a closed conformation. Structure-guided mutagenesis confirmed critical interactions between the SMRT peptides and HDAC4 and -5 as well as the contribution of the Arg1369 residue in the first motif for optimal binding to the two HDACs. These results indicate that SMRT binding does not activate the cryptic deacetylase activity of HDAC4 and explain how class IIa HDACs and the SMRT-HDAC3 complex are coordinated during gene regulation.