Defects in spermatogenesis, a process that produces spermatozoa inside seminiferous tubules of the testis, result in male infertility. Spermatogenic progression is highly dependent on a microenvironment provided by Sertoli cells, the only somatic cells and epithelium of seminiferous tubules. However, genes that regulate such an important activity of Sertoli cells are poorly understood. Here, we found that AT-rich interactive domain 4B (ARID4B), is essential for the function of Sertoli cells to regulate spermatogenesis. Specifically, we generated Sertoli cell-specific Arid4b knockout (Arid4bSCKO) mice, and showed that the Arid4bSCKO male mice were completely infertile with impaired testis development and significantly reduced testis size. Importantly, severe structural defects accompanied by loss of germ cells and Sertoli cell-only phenotype were found in many seminiferous tubules of the Arid4bSCKO testes. In addition, maturation of Sertoli cells was significantly delayed in the Arid4bSCKO mice, associated with delayed onset of spermatogenesis. Spermatogenic progression was also defective, showing an arrest at the round spermatid stage in the Arid4bSCKO testes. Interestingly, we showed that ARID4B functions as a "coactivator" of androgen receptor and is required for optimal transcriptional activation of reproductive homeobox 5, an androgen receptor target gene specifically expressed in Sertoli cells and critical for spermatogenesis. Together, our study identified ARID4B to be a key regulator of Sertoli cell function important for male germ cell development.