Intuitively, light impinging on a spatially mirror-symmetric object will be scattered equally into mirror-symmetric directions. This intuition can fail at the nanoscale if the polarization of the incoming light is properly tailored, as long as mirror symmetry is broken in the axes perpendicular to both the incident wave vector and the remaining mirror-symmetric direction. The unidirectional excitation of plasmonic modes using circularly polarized light has been recently demonstrated. Here, we generalize this concept and show that linearly polarized photons impinging on a single spatially symmetric scatterer created in a silicon waveguide are guided into a certain direction of the waveguide depending exclusively on their polarization angle and the structure asymmetry. Our work broadens the scope of polarization-induced directionality beyond plasmonics, with applications in polarization (de)multiplexing, unidirectional coupling, directional switching, radiation polarization control, and polarization-encoded quantum information processing in photonic integrated circuits.