Personalized tumor growth model using clinical imaging data is valuable in tumor staging and therapy planning. In this paper, we build a patient specific tumor growth model based on longitudinal dual phase CT and FDG-PET. We propose a reaction-advection-diffusion model integrating cancerous cell proliferation, infiltration, metabolic rate and extracellular matrix biomechanical response. We then develop a scheme to bridge our model with multimodal radiologic images through intracellular volume fraction (ICVF) and Standardized Uptake Value (SUV). The model was evaluated by comparing the predicted tumors with the observed tumors in terms of average surface distance (ASD), root mean square difference (RMSD) of the ICVF map, the average ICVF difference (AICVFD) of tumor surface and the tumor relative volume difference (RVD) on six patients with pathologically confirmed pancreatic neuroendocrine tumors. The ASD between the predicted tumor and the reference tumor was 2.5 +/- 0.7 mm, the RMSD was 4.3 +/- 0.6%, the AICVFD was 2.6 +/- 0.8%, and the RVD was 7.7 +/- 1.9%.