We report circular-to-linear and linear-to-circular conversion of optical polarization by semiconductor quantum dots. The polarization conversion occurs under continuous wave excitation in the absence of any magnetic field. The effect originates from quantum interference of linearly and circularly polarized photon states, induced by the natural anisotropic shape of the self-assembled dots. The behavior can be qualitatively explained in terms of a pseudospin formalism.