Recently, an optical scanning holographic system with a polarization directed flat lens was proposed to realize coaxial scanning holography (CSH). The advantage of CSH is its small form factor and the stability. However, the diffraction efficiency of the polarization directed flat lens cannot be 100%, and thus there is always zeroth order light in the scanning beam. The imperfect diffraction property of the polarization directed flat lens results in an incomplete scanning Fresnel zone plate. Consequently, the reconstructed image is blurred and noisy. In this paper, we compared different methods, including the back propagation, the phase correlation, and inverse filtering, for the hologram reconstruction. It is demonstrated that inverse filtering is the only method that can retrieve the high-frequency component of the hologram. However, additional noise also arises with the use of inverse filtering. Therefore, the imaging performance of CSH by using a polarization directed flat lens is inherently worse than that of conventional OSH.