The genome-wide variation of multiple epigenetic modifications in CpG islands (CGIs) and the interactions between them are of great interest. Here, we optimized an entropy-based strategy to quantify variation of epigenetic modifications and explored their interaction across mouse embryonic stem cells, neural precursor cells and brain. Our results showed that four epigenetic modifications (DNA methylation, H3K4me2, H3K4me3 and H3K27me3) of CGIs in the mouse genome undergo combinatorial variation during neuron differentiation. DNA methylation variation was positively correlated with H3K27me3 variation, and negatively correlated with H3K4me2/3 variation. We identified 5,194 CGIs differentially modified by epigenetic modifications (DEM-CGIs). Among them, the differentially DNA methylated CGIs overlapped significantly with the CGIs differentially modified by H3K27me3. Moreover, DEM-CGIs may contribute to co-regulation of related developmental genes including core transcription factors. Our entropy-based strategy provides an effective way of investigating dynamic cross-talk among epigenetic modifications in various biological processes at the macro scale.