In this paper, a three-layered sandwiched metamaterial is proposed to achieve giant and broadband asymmetric transmission of circularly polarized waves at the near-infrared communication band. The metamaterial consists of two layers of identical 45° tilted chiral S-shaped metasurfaces sandwiched with a subwavelength metallic grating. Based on the delicate combination of broadband polarization conversion and a cavity-enhanced effect, the asymmetric parameter can reach a maximum value of 0.87 and over 0.6 in a wide range from 1.2 to 2.0 μm, which has not been found in previous reports. Furthermore, a perfect robustness to misalignments is obtained as the effect originated from function-independent cascading cavities, which effectively reduce the requirement of alignment precision in layer-by-layer photolithography processes. The proposed nanostructure has a great potential to be used as a circular polarization rotator or diode-like device in optical communication systems.