Plasmonic-polymer nanocomposites can serve as a multifunctional platform for a wide range of applications such as biochemical sensing and photothermal treatments, where they synergistically benefit from the extraordinary optical properties of plasmonic nanoparticles (NPs) and biocompatible characteristics of biopolymers. The field translation of plasmonic-polymer nanocomposites requires design rules for scalable and reproducible fabrication with tunable and predictable optical properties and achieving the best performance. The optical properties of NPs and the optimal analytical performance of nanocomposites could be affected by many fabrication parameters, but a fundamental understanding of such parameters is still minimal. Herein, we systematically investigated the NP distribution and their optical properties in gold nanostar (GNS)-polymer nanocomposites as a function of GNS concentration, polymer identity, and the method of GNS incorporation into a polymer matrix. We performed a comprehensive analysis of the single-particle scattering spectra of GNS incorporated into agarose gel and chitosan hydrogels via embedding and surface deposition, using dark-field spectroscopy. While relative GNS concentration affects the GNS scattering property distribution in both polymer matrices, chemical interactions between a polymer matrix and GNS is the key determinant of the GNS stability and homogenous distribution in nanocomposites. When GNS are embedded in a polymer matrix and there are stronger chemical interactions between GNS and a polymer, significantly less aggregation and a more homogenous distribution of GNS, which leads to a larger percentage of GNS optical property preservation, were observed at all the concentrations. In a proof-of-concept surface-enhanced Raman spectroscopy (SERS) study, we observed that the SERS detection efficiency is dictated by the analyte accessibility of GNS, which is governed by the polymer matrix porosity, polymer-GNS interactions, and other polymer physical characteristics. This work presents the interplay between key fabrication parameters and foundational design parameters for more predictable and reliable fabrication of plasmonic-polymer nanocomposites as an optical platform.