A nanoscale layer of chromium or titanium is commonly used in plasmonic nanoantennas to firmly adhere a gold film to a glass substrate, yet the influence of this layer on the antenna performance is often ignored. As a result, the need for the use of potentially better materials is not widely recognized. Using a single aperture milled in a gold film with 120 nm diameter as a nanobench for these investigations, we present the first experimental report of the strong dependence of the plasmonic enhancement of single-molecule fluorescence on the nature of the adhesion layer. By combining fluorescence correlation spectroscopy and fluorescence lifetime measurements, we show that this structure is very sensitive to the properties of the adhesion layer, and we detail the respective contributions of excitation and emission gains to the observed enhanced fluorescence. Any increase in the absorption losses due to the adhesion layer permittivity or thickness is shown to lower the gains in both excitation and emission, which we relate to a damping of the energy coupling at the nanoaperture. With this nanobench, we demonstrate the largest enhancement factor reported to date (25×) by using a TiO(2) adhesion layer. The experimental data are supported by numerical simulations and argue for a careful consideration of the adhesion layer while designing nanoantennas for high-efficiency single-molecule analysis.