Most homodyne interferometers have a quadrature detector system that includes two polarizing beam splitters that cause nonlinearity of the order of a few nanometers by phase mixing. Detectors should have the same gains to reduce nonlinearity under the assumption that there is no loss in optical components. However, optical components exhibit some loss. We show that nonlinearity can be reduced to an order of 0.01 nm when the detector gains are adjusted by simulation to include the optical characteristics. The compensated nonlinearity is 18 times smaller than that when the four detector gains are set to be equal.