In Fourier domain optical coherence tomography (FDOCT), the depth profile is mirrored about the zero delay between the sample and reference optical paths, limiting the imaging depth to half of the entire ranging space and undermining the optimal sensitivity window. We present a new method, to the best of our knowledge, to remove the complex conjugate artifact by using circularly polarized light as reference. Quadrature detection of the complex fringe is achieved by utilizing the intrinsic λ/4 delay between two polarization channels. We use passive broadband polarization optics to control the polarization state of the light in the reference and sample arms and a balanced polarization diversity detection unit to simultaneously detect phase-shifted fringes. We demonstrate a 40 dB artifact suppression ratio with a swept-source optical coherence tomography system. Our proposed method is immune to sample motion and laser phase noise, and imposes no restrictions to the source bandwidth, imaging speed, or computational power. In vivo images of the human finger, as well as the cornea and retina of a non-human primate, were demonstrated.