We investigate the optical properties of mono- and double-layer asymmetric fishnet metamaterials with orientated elliptical holes, which exhibit exotic spectral and polarization rotating characteristics in the visible spectral range. Our results show that nontrivial orientations of the holes with respect to the reciprocal lattice vectors of the periodic lattice in both systems produce strong polarization rotation as well as additional enhanced optical transmission peaks. Analysis of the electromagnetic field distribution shows the unusual effect is produced by the spinning localized surface plasmon resonances due to the asymmetric geometry. High sensitivity of the hybridized mode on the dielectric spacing, the aspect ratio of the holes and the embedding media in double-layer structure is also observed. The dependence of spectral and polarization response on the orientation of the holes and the embedding media is useful for design of chiral metamaterials at optical frequencies and tailoring the polarization behavior of the metallic nano-structures.