The polarization-sensitive imaging technology is proposed based on incoherent holography. The distribution of state of polarization (SoP) of the object light field can be reconstructed by measuring the phase difference and amplitude ratio of two components of the Jones vector on the basis of incoherent self-interference theory and the accurate point spread function (PSF) of the incoherent holographic system. In the analysis of Fresnel diffraction, we develop a new method to greatly simplify the calculation of the accurate PSF by means of imaging property of lens and symbolic mathematics tools. In the recording process, we utilize the automation of phase shift, photography, and synthesization of color hologram to greatly shorten the total recording time of a group of phase-shifted holograms. The experimental results show that the proposed technology can accurately realize polarization-sensitive imaging and it is much simpler for complete linearly polarized light.