Surface-enhanced Raman spectroscopy (SERS) is considered a highly promising technology for different analytical purposes. The applications of SERS are still quite limited due its relatively poor quantitative repeatability and the fact that SERS is very sensitive to oxidation, which is a challenge especially with silver based SERS substrates. Here, the link between these phenomena is investigated by exposing silver SERS substrates to ambient laboratory air. We show that SERS intensity decreases exponentially after the exposure, which consequently leads to an increasing standard deviation (σ) in intensity. Within a five-hour measurement window, the SERS intensity already drops by 60%, while σ triples from 7% to 21%. The SERS results are supplemented by elemental analysis, which shows that oxidation and atmospheric carbon contamination coincide with the rapid SERS intensity decrease. The results emphasize how sensitive SERS is towards atmospheric contamination and how it can also reduce the measurement repeatability - even if the substrates are exposed to air just for a very short period of time.