A new cascaded modulator structure that has the ability to realize high conversion efficiency microwave frequency downconversion, while at the same time able to overcome two fundamental limitations in the dual-parallel modulator approach, is presented. It is based on utilizing the polarization-dependent modulation efficiency property in LiNbO3 electro-optic modulators. The new structure allows the modulators for the RF signal and local oscillator (LO) modulation to be placed in different locations suitable for antenna remoting applications, and it has infinite isolation between the LO and RF signal ports. We present experimental results demonstrating that the proposed structure can be used to realize high conversion efficiency frequency downconversion over wide RF and intermediate frequency (IF) signal frequency ranges as the reported dual-parallel-modulator-based microwave photonic frequency downconverter. Very high isolation of more than 70 dB between the LO and RF signal ports is also demonstrated.