Pancreatic neuroendocrine tumors are abnormal growths of hormone-producing cells in the pancreas. Different from the brain in the skull, the pancreas in the abdomen can be largely deformed by the body posture and the surrounding organs. In consequence, both tumor growth and pancreatic motion attribute to the tumor shape difference observable from images. As images at different time points are used to personalize the tumor growth model, the prediction accuracy may be reduced if such motion is ignored. Therefore, we incorporate the image-derived pancreatic motion to tumor growth personalization. For realistic mechanical interactions, the multiplicative growth decomposition is used with a hyperelastic constitutive law to model tumor mass effect, which allows growth modeling without compromising the mechanical accuracy. With also the FDG-PET and contrast-enhanced CT images, the functional, structural, and motion data are combined for a more patient-specific model. Experiments on synthetic and clinical data show the importance of image-derived motion on estimating physiologically plausible mechanical properties and the promising performance of our framework. From six patient data sets, the recall, precision, Dice coefficient, relative volume difference, and average surface distance were 89.8 ± 3.5%, 85.6 ± 7.5%, 87.4 ± 3.6%, 9.7 ± 7.2%, and 0.6 ± 0.2 mm, respectively.