Pulmonary Embolism (PE) is a severe medical condition that can pose a significant risk to life. Traditional deep learning methods for PE diagnosis are based on Computed Tomography (CT) images and do not consider the patient's clinical context. To make full use of patient's clinical information, this article presents a multimodal fusion model ingesting Electronic Health Record (EHR) data and CT images for PE diagnosis. The proposed model is based on multilayer perception and convolutional neural networks. To remove the invalid information in the EHR data, the multidimensional scaling algorithm is performed for feature dimension reduction. The EHR data and CT images of 600 patients are used for experiments. The experiment results show that the proposed models outperform existing methods and the multimodal fusion model shows better performance than the single-input model.