Measurement-device-independent quantum key distribution (MDI-QKD) with the active decoy state method can remove all detector loopholes, and resist the imperfections of sources. But it may lead to side channel attacks and break the security of QKD system. In this paper, we apply the passive decoy state method to the MDI-QKD based on polarization encoding mode. Not only all attacks on detectors can be removed, but also the side channel attacks on sources can be overcome. We get that the MDI-QKD with our passive decoy state method can have a performance comparable to the protocol with the active decoy state method. To fit for the demand of practical application, we discuss intensity fluctuation in the security analysis of MDI-QKD protocol using passive decoy state method, and derive the key generation rate for our protocol with intensity fluctuation. It shows that intensity fluctuation has an adverse effect on the key generation rate which is non-negligible, especially in the case of small data size of total transmitting signals and long distance transmission. We give specific simulations on the relationship between intensity fluctuation and the key generation rate. Furthermore, the statistical fluctuation due to the finite length of data is also taken into account.