The development and function of the brain require tight control of gene expression. Genome architecture is thought to play a critical regulatory role in gene expression, but the mechanisms governing genome architecture in the brain in vivo remain poorly understood. Here, we report that conditional knockout of the chromatin remodeling enzyme Chd4 in granule neurons of the mouse cerebellum increases accessibility of gene regulatory sites genome-wide in vivo. Conditional knockout of Chd4 promotes recruitment of the architectural protein complex cohesin preferentially to gene enhancers in granule neurons in vivo. Importantly, in vivo profiling of genome architecture reveals that conditional knockout of Chd4 strengthens interactions among developmentally repressed contact domains as well as genomic loops in a manner that tightly correlates with increased accessibility, enhancer activity, and cohesin occupancy at these sites. Collectively, our findings define a role for chromatin remodeling in the control of genome architecture organization in the mammalian brain.