Chiral plasmonic structures have been shown to possess large circular dichroism (CD) responses. Here, we investigate the CD responses in a solid and inverse metallic structure composed of a stacked right-twisted gammadion metallic nanoparticle and a left-twisted gammadion nanoaperture array, where a giant circular dichroism is achieved. In addition, the sign of the CD responses can be reversed through the changes of the geometric parameters. Further analysis reveals that the Fabry-Perot (F-P) resonance of cross-polarization conversion of electric field governs the change of the CD. It can be envisioned that our findings will allow further tuning and manipulation of the CD responses for tailored circular polarized light-matter interaction.