Using inelastic electron scattering in combination with dielectric theory simulations on differently prepared graphene layers on silicon carbide, we demonstrate that the coupling between the 2D plasmon of graphene and the surface optical phonon of the substrate cannot be quenched by modification of the interface via intercalation. The intercalation rather provides additional modes like, e.g., the silicon-hydrogen stretch mode in the case of hydrogen intercalation or the silicon-oxygen vibrations for water intercalation that couple to the 2D plasmons of graphene. Furthermore, in the case of bilayer graphene with broken inversion symmetry due to charge imbalance between the layers, we observe a similar coupling of the 2D plasmon to an internal infrared-active mode, the LO phonon mode. The coupling of graphene plasmons to vibrational modes of the substrate surface and internal infrared active modes is envisioned to provide an excellent tool for tailoring the plasmon band structure of monolayer and bilayer graphene for plasmonic devices such as plasmon filters or plasmonic waveguides. The rigidity of the effect furthermore suggests that it may be of importance for other 2D materials as well.