We demonstrate a passive quadrature demodulation method using polarized light and a fiber-optic sensor with a birefringent low-finesse Fabry-Perot (FP) cavity. With precisely controlled birefringence in the FP cavity, the fringes probed by light polarized along the two principal axes can have a quadrature phase shift and can be separately detected. We demonstrated the concept for ultrasound detection using a sensor with an FP cavity formed by two low-reflectivity chirped fiber Bragg gratings on a coiled polarization-maintaining (PM) fiber. By controlling the bending radius, bending length, and twist of the coil structure, we obtained an optimized total phase delay of 109° between the fringes of the two polarizations along the principal axes of the PM fiber. Using a linearly polarized laser source and a polarization beam splitter, the signals at the two polarizations were separated and measured by two photodetectors. The experimental results show that the sensor can detect ultrasonic signal when the sensor spectra experience environmental drifts using a laser at a fixed wavelength.