We propose a comprehensive point spread function (PSF) degradation model, which considers multiple factors consisting of degradation of specimen retardant sampling and polarization angularly anamorphic sampling, to indicate the image degradation characteristics of polarization imaging systems. First, a one-layer optical coherence tomography (OCT) model was established to express the retardancy of medium-loading specimens. Then, a PSF degradation model of angularly anamorphic polarization sampling was deduced through the retrieval of Stokes parameters. Finally, maximum a posteriori probability (MAP) was adopted to assess the distribution of the proposed model. Hypothesis testing using actual data and numerical simulations demonstrated that the error of the system followed an asymmetric generalized Gaussian distribution (AGGD). Finite-difference time-domain (FDTD) simulation results and an actual imaging experiment demonstrate the consistency of the proposed model and the degradation characteristics of the PSF, which provide support for the improved accuracy and enhanced image quality of the optical field retrieval of nanoparticles.