The performance of plasmon in applications is strongly related to plasmon damping, i.e., a dephasing of the optical polarization associated with the electron oscillation. Accurate measurement, manipulation, and, ultimately, prolongation of the dephasing time are prerequisites to the future development of the application of plasmonics. Here, we studied the dephasing time of different plasmonic hotspots in an individual bowtie structure by time-resolved photoemission electron microscopy and proposed an easy-to-operate method for actively and flexibly controlling the mode-dependent plasmon dephasing time by varying the polarization direction of a femtosecond laser. Experimentally, we achieved a large adjustment of the dephasing time ranging from 7 to 17 fs. In addition, a structural defect was found to drastically extend the plasmon dephasing time. Assisted with the finite-difference time-domain simulation, the underlying physics of the dephasing time extension by the structural defect was given.