Light propagation wavefront and photon composition variations occur when the beam encounters acoustic waves, bringing mechanical and chemical inhomogeneity-induced light-intensity modulation, while phase variations, which carry more information about the acoustic-optical coupling in the medium, are often overlooked. This paper investigates the coupling of the light beam with the propagating ultrasound and the polarization aberration of the optical wave induced by the ultrasound. A model was developed to express the variation of the ultrasound-induced polarization aberration (UIPA). The ultrasound-induced refractive index variation of the sample was observed in both the simulation and experiments. The phase differences in various ultrasound states (valley dominant state, peak dominant state) are characterized in detail. The UIPA expressed in the phase space provides a way to quantify multidimensional polarization information of the ultrasound-tagged optical waves and allows refraction-sensitive polarization parametric imaging, which may be exploited for directional high-contrast photoacoustic imaging with ultrasound tagging.