Using three-dimensional (3D) coherency vector (9 × 1), we develop a new 3D polarization algebra to calculate the polarization properties of all polarization sensitive optical systems, especially when the incident optical field is partially polarized or un-polarized. The polarization properties of a high numerical aperture (NA) microscope objective (NA = 1.25 immersed in oil) are analyzed based on the proposed 3D polarization algebra. Correspondingly, the polarization simulation of this high NA optical system is performed by the commercial software VirtualLAB Fusion. By comparing the theoretical calculations with polarization simulations, a perfect matching relation is obtained, which demonstrates that this 3D polarization algebra is valid to quantify the 3D polarization properties for all polarization sensitive optical systems.