Histone H3 lysine 4 trimethylation (H3K4me3) is a hallmark of transcription initiation, but how H3K4me3 is demethylated during gene repression is poorly understood. Jhd2, a JmjC domain protein, was recently identified as the major H3K4me3 histone demethylase (HDM) in Saccharomyces cerevisiae. Although JHD2 is required for removal of methylation upon gene repression, deletion of JHD2 does not result in increased levels of H3K4me3 in bulk histones, indicating that this HDM is unable to demethylate histones during steady-state conditions. In this study, we showed that this was due to the negative regulation of Jhd2 activity by histone H3 lysine 14 acetylation (H3K14ac), which colocalizes with H3K4me3 across the yeast genome. We demonstrated that loss of the histone H3-specific acetyltransferases (HATs) resulted in genome-wide depletion of H3K4me3, and this was not due to a transcription defect. Moreover, H3K4me3 levels were reestablished in HAT mutants following loss of JHD2, which suggested that H3-specific HATs and Jhd2 serve opposing functions in regulating H3K4me3 levels. We revealed the molecular basis for this suppression by demonstrating that H3K14ac negatively regulated Jhd2 demethylase activity on an acetylated peptide in vitro. These results revealed the existence of a general mechanism for removal of H3K4me3 following gene repression.