Based on Bethe's theory, light is hard to transmit through sub-wavelength apertures. However, a special designed sub-wavelength bowtie aperture is found to be able to transmit light with high efficiency. In this letter, modal analysis is used to study the hybridized plasmonic modes and Fabry-Perot effect of the nanoscale bowtie aperture waveguide. High frequency structure simulator (HFSS) simulations in perfect electrically conductor (PEC) and real metals are performed to calculate the fundamental mode, higher order mode, as well as their own cutoff wavelength. Mode analysis can give a better understanding of the intrinsic link between the plasmonic effects and Fabry-Perot effect. The TE10 and TE30 modes hybridize with channel plasmon polaritons (CPPs) modes and surface plasmon polaritons (SPPs) modes respectively. Experiments are carried out to verify the numerical results. These results are of great significance for understanding the internal mechanism of the bowtie aperture for coupling light to a sub-diffraction limited spot with high transmission efficiency.