Plasmonic nanoclusters can strongly absorb light energy and generate hot carriers, which have great potentials in photovoltaic and photocatalytic applications. A vital step for those plasmonic applications is the charge transfer at the metal-semiconductor interface. The effect of the light polarization on the charge transfer has not been theoretically investigated so far. Here, we take the Ag-TiO2 system as a model system to study the polarization effect using time-dependent density functional theory simulations. We find that the charge transfer is sensitive to the light polarization, which has its origin in the polarization-dependent hot carrier distributions. For the linearly polarized light, it shows a sine-square dependence on the polar angle, indicating that the charge transfer response to the linear polarization can be decomposed into components perpendicular and parallel to the interface. We also find that there exists directional charge transfer with a circular light polarization. Our results demonstrate that the light polarization can significantly affect the charge transfer behavior and, thus, offer a new degree of freedom to manipulate the plasmonic applications.