Generally, polarization and phase are considered as two relatively independent parameters of light, and show little interaction when a light propagates in a homogeneous and isotropic medium. Here, we reveal that orbital angular momentum (OAM) of an optical vortex beam can be modulated by specially-tailored locally linear polarization states of light under a tightly-focusing conditon. We perform both theoretical and experimental studies of this interaction between vortex phase and vector polarization, and find that an arbitrary topological charge value of OAM can be achieved in principle through vector polarization modulation, in contrast to the spin-orbital conversion that yields only the ± ћ OAM values through circular polarization. We verify the modulation of optical OAM state with vector beams by observing the orbital rotation of trapped particles.