In this Letter we report a comparative study, in the infrared regime, of surface plasmon polariton (SPP) propagation in epitaxially grown Ag films and in polycrystalline Ag films, all grown on Si substrates. Plasmonic resonance features are analyzed using extraordinary optical transmission (EOT) measurements, and SPP band structures for the two dielectric/metal interfaces are investigated for both types of film. At the Si/Ag interface, EOT spectra show almost identical features for epitaxial and polycrystalline Ag films and are characterized by sharp Fano resonances. On the contrary, at the air/Ag interface, dramatic differences are observed: while the epitaxial film continues to exhibit sharp Fano resonances, the polycrystalline film shows only broad spectral features and much lower transmission intensities. In corroboration with theoretical simulations, we find that surface roughness plays a critical role in SPP propagation for this wavelength range.