Active manipulation of the polarization states at terahertz frequencies is crucially helpful for polarization-sensitive spectroscopy, having significant applications such as non-contact Hall measurements, vibrational circular dichroism measurements and anisotropy imaging. The weakness of polarization manipulation provided by natural materials can be overcomed by chiral metamaterials. Chiral metamaterials have a huge potential to achieve the necessary polarization effects, hence they provide the basis for applications such as ultracompact polarization components. Terahertz chiral metamaterials that allow dynamic polarization modulation of terahertz waves are of great practical interest and still challenging. Here, we show that terahertz metasurface based on the four conjugated "petal" resonators integrated with multi-layered graphene (MLG) can enable dynamically tunable chiroptical response using optical pumping. In particular, a change of ellipticity angle of 20° is observed around 0.76 THz under optical pumping by a 980 nm continuous wave (CW) laser. Furthermore, using temporal coupled-mode theory, our study also reveals that the chiroptical response of the proposed multi-layered graphene-based metasurface is strongly dependent on the influence of optical pumping on the loss parameters of resonance modes, leading to actively controllable polarization states of the transmitted terahertz waves. The present work paves the way for the realization of fundamental terahertz components capable for active polarization manipulation.