In this paper, we demonstrate the combination of a dielectric metasurface with a graphene layer to realize a high performance toroidal resonance based optical modulator. The dielectric metasurface consists of two mirrored asymmetric silicon split-ring resonators (ASSRRs) that can support strong toroidal dipolar resonance with narrow line width (~0.77 nm) and high quality (Q)-factor (~1702) and contrast ratio (~100%). Numerical simulation results show that the transmission amplitude of the toroidal dipolar resonance can be efficiently modulated by varying the Fermi energy EF when the graphene layer is integrated with the dielectric metasurface, and a max transmission coefficient difference up to 78% is achieved indicating that the proposed hybrid graphene/dielectric metasurface shows good performance as an optical modulator. The effects of the asymmetry degree of the ASSRRs on the toroidal dipolar resonance are studied and the efficiency of the transmission amplitude modulation of graphene is also investigated. Our results may also provide potential applications in optical filter and bio-chemical sensing.