Surface-enhanced Raman scattering (SERS) spectroscopy has attracted tremendous interests as a highly sensitive label-free tool. The local field produced by the excitation of localized surface plasmon resonances (LSPRs) dominates the overall enhancement of SERS. Such an electromagnetic enhancement is unfortunately accompanied by a strong modification in the relative intensity of the original Raman spectra, which highly distorts spectral features providing chemical information. Here we propose a robust method to retrieve the fingerprint of intrinsic chemical information from the SERS spectra. The method is established based on the finding that the SERS background originates from the LSPR-modulated photoluminescence, which contains the local field information shared also by SERS. We validate this concept of retrieval of intrinsic fingerprint information in well controlled single metallic nanoantennas of varying aspect ratios. We further demonstrate its unambiguity and generality in more complicated systems of tip-enhanced Raman spectroscopy (TERS) and SERS of silver nanoaggregates.