We used a combination of Raman microscopy, AFM and TEM to quantify the influence of dimerization on the surface enhanced Raman spectroscopy (SERS) signal for gold and silver nanoparticles (NPs) modified with Raman reporters and situated on gold, silver, and aluminum films and a silicon wafer. The overall increases in the mean SERS enhancement factor (EF) upon dimerization (up by 43% on average) and trimerisation (up by 96% on average) of AuNPs and AgNPs on the studied metal films are within a factor of two, which is moderate when compared to most theoretical models. However, the maximum ratio of EFs for some dimers to the mean EF of monomers can be as high as 5.5 for AgNPs on a gold substrate. In contrast, for dimerization and trimerization of gold and silver NPs on silicon, the mean EF increases by 1-2 orders of magnitude relative to the mean EF of single NPs. Therefore, hot spots in the interparticle gap between gold nanoparticles rather than hot spots between Au nanoparticles and the substrate dominate SERS enhancement for dimers and trimers on a silicon substrate. However, Raman labeled noble metal nanoparticles on plasmonic metal films generate on average SERS enhancement of the same order of magnitude for both types of hot spot zones (e.g. NP/NP and NP/metal film).