We study the polarization properties of quantum dot (QD) emission coupled with the fundamental cavity modes. A rotation of polarization axis and a change of polarization degree are observed as the coupling is varied. To explain this observation, we derive an analytical model considering the polarization misalignment between QD dipole and cavity mode field. Our model also provides a new approach to extract the anisotropic Purcell factors by analyzing the polarization of detected quantum dot emission coupled to the cavity mode, which paves the way to develop high-efficiency polarized single photon sources.