Chinese herbal medicines (CHMs) derived from nature have received increasing attention and become more popular. Due to their diverse production processes, complex ingredients, and different storage conditions, it is highly desirable to develop simple, rapid, efficient and trace detection methods to ensure the drug quality. Surface-enhanced Raman spectroscopy has the advantages of being time-saving, non-destructive, usable in aqueous environments, and highly compatible with various biomolecular samples, providing a promising analytical method for CHM. In this review, we outline the major advances in the application of SERS to the identification of raw materials, detection of bioactive constituents, characterization of adulterants, and detection of contaminants. This clearly shows that SERS has strong potential in the quality control of CHM, which greatly promotes the modernization of CHM.