The gene encoding the transcription factor TFAP4/AP4 represents a direct target of the c-MYC oncoprotein. Here, we deleted Ap4 in ApcMin mice, a preclinical model of inherited colorectal cancer. Ap4 deficiency extends their average survival by 110 days and decreases the formation of intestinal adenomas and tumor-derived organoids. The effects of Ap4 deletion are presumably due to the reduced number of functional intestinal stem cells (ISCs) amenable to adenoma-initiating mutational events. Deletion of Ap4 also decreases the number of colonic stem cells and increases the number of Paneth cells. Expression profiling revealed that ISC signatures, as well as the Wnt/β-catenin and Notch signaling pathways are downregulated in Ap4-deficient adenomas and intestinal organoids. AP4-associated signatures are conserved between murine adenomas and human colorectal cancer samples. Our results establish Ap4 as rate-limiting mediator of adenoma initiation, as well as regulator of intestinal and colonic stem cell and Paneth cell homeostasis.