In this work, we investigate the coupling of surface plasmons and excited optical modes in metal/dielectric grating stacks theoretically and experimentally. We have observed three kinds of modes in these structures: the cavity mode, the propagated surface plasmon (PSP) mode and the localized surface plasmon (LSP) mode, which can enhance the optical transmission. Firstly, it is shown that the cavity mode is excited in the grating stacks. And the cavity mode has redshift if we enhance the thickness of metal layers, while it has blueshift when we increase the thickness of dielectric layers. The redshift of the cavity mode also occurs when the number of repeating layers is increased. Secondly, the PSP mode is also excited, which can be described by the effective permittivity method. It is found that the PSP modes are coupled with each other, which leads to a modified dispersion relation of surface plasmon polaritons (SPP). The theoretical analysis is in good agreement with the observed transmission enhancement in the grating stacks. And the coupling of PSPs also leads to a blueshift when the number of metal layers is increased. Thirdly, the LSP mode, generated in single metal strip, can also enhance the optical transmission of the grating stacks. Yet the transmission intensity induced by LSP decreases rapidly with increasing the number of metal layers. The investigations here may have potential applications in designing plasmonic metamaterials and subwavelength optical devices.