MRI-guided proton therapy is being pursued for its promise to provide a more conformal, accurate proton therapy. However, the presence of the magnetic field imposes a challenge for the beam delivery as protons are deflected due to the Lorenz force. In this study, the impact of realistic inline MRI fringe field on IMPT plan delivery is investigated for a water phantom, liver tumor and prostate cancer differing in target volume, shape, and field configuration using Monte Carlo simulations. A method to correct for the shift of the beam spot positions in the presence of the inline magnetic field is presented. Results show that when not accounting for the effect of the magnetic field on the pencil beam delivery, the spot positions are substantially shifted and the quality of delivered plans is significantly deteriorated leading to dose inhomogeneities and creation of hot and cold spots. However, by correcting the pencil beam delivery, the dose quality of the IMPT plans is restored to a high degree. Nevertheless, adaptation of beam delivery alone is not robust regarding different treatment sites. By fully accounting during plan optimization for the dose distortions caused by the fringe and imaging fields, highly conformal IMPT plans are achieved. These results demonstrate proton pencil beam scanning and treatment planning can be adapted for precise delivery of state-of-the-art IMPT plans in MR-guided proton therapy in the presence of an inline MRI fringe field.