Rapidly and accurately detecting and quantifying the concentrations of nitroaromatic explosives is critical for public health and security. Among existing approaches, explosives' detection with Surface-Enhanced Raman Spectroscopy (SERS) has received considerable attention due to its high sensitivity. Typically, a preprocessed single spectrum that is the average of the entire or a selected subset of a SERS map is used to train various machine learning models for detection and quantification. Designing an appropriate averaging and preprocessing procedure for SERS maps across different concentrations is time-consuming and computationally costly, and the averaging of spectra may lead to the loss of crucial spectral information. We propose an attention-based vision transformer neural network for nitroaromatic explosives' detection and quantification that takes raw SERS maps as the input without any preprocessing. We produce two novel SERS datasets, 2,4-dinitrophenols (DNP) and picric acid (PA), and one benchmark SERS dataset, 4-nitrobenzenethiol (4-NBT), which have repeated measurements down to concentrations of 1 nM to illustrate the detection limit. We experimentally show that our approach outperforms or is on par with the existing methods in terms of detection and concentration prediction accuracy. With the produced attention maps, we can further identify the regions with a higher signal-to-noise ratio in the SERS maps. Based on our findings, the molecule of interest detection and concentration prediction using raw SERS maps is a promising alternative to existing approaches.