Adhesion layers, required to stabilize metallic nanostructures, dramatically deteriorate the performances of plasmonic sensors, by severely damping the plasmon modes. In this article, we show that these detrimental effects critically depend on the overlap of the electromagnetic near-field of the resonant plasmon mode with the adhesion layer and can be minimized by careful engineering of the latter. We study the dependence of the geometrical parameters such as layer thickness and shape on the near-field of localized plasmon resonances for traditional adhesion layers such as Cr, Ti, and TiO2. Our experiments and simulations reveal a strong dependence of the damping on the layer thickness, in agreement with the exponential decay of the plasmon near-field. We developed a method to minimize the damping by selective deposition of thin adhesion layers (<1 nm) in a manner that prevents the layer to overlap with the hotspots of the plasmonic structure. Such a designed structure enables the use of standard Cr and Ti adhesion materials to fabricate robust plasmonic sensors without deteriorating their sensitivity.