We demonstrate optical frequency conversion between telecom wavelengths using four-wave mixing Bragg scattering powered by two pump pulses polarized on orthogonal axes of a silicon waveguide. This allows conversion in a single frequency direction while, with co-polarized pumps, the signal is redshifted or blueshifted with similar efficiency. Our approach exploits the birefringence of the waveguide and its effect on the phase matching of the four-wave mixing process. The blue or red direction can be selected by the input polarization of the signal, and 20 dB extinction ratios are observed with the unintended direction. This technique will allow efficient and controlled conversion between specified wavelength channels in integrated photonic devices.