DNA methyltransferases (DNMTs), responsible for the regulation of DNA methylation, have been regarded as promising drug targets for cancer therapy. However, high structural conservation of the catalytic domains of DNMTs poses a big challenge to design selective inhibitors for a specific DNMT isoform. In this study, molecular dynamics (MD) simulations, end-point free energy calculations and umbrella sampling (US) simulations were performed to reveal the molecular basis of the binding selectivity of three representative DNMT inhibitors towards DNMT1 and DNMT3A, including SFG (DNMT1 and DNMT3A dual inhibitors), DC-05 (DNMT1 selective inhibitor) and GSKex1 (DNMT3A selective inhibitor). The binding selectivity of the studied inhibitors reported in previous experiments is reproduced by the MD simulation and binding free energy prediction. The simulation results also suggest that the driving force to determine the binding selectivity of the studied inhibitors stems from the difference in the protein-inhibitor van der Waals interactions. Meanwhile, the per-residue free energy decomposition reveals that the contributions from several non-conserved residues in the binding pocket of DNMT1/DNMT3A, especially Val1580/Trp893, Asn1578/Arg891 and Met1169/Val665, are the key factors responsible for the binding selectivity of DNMT inhibitors. In addition, the binding preference of the studied inhibitors was further validated by the potentials of mean force predicted by the US simulations. This study will provide valuable information for the rational design of novel selective inhibitors targeting DNMT1 and DNMT3A.