We present an effective scheme to improve the modulation capacity in graphene-based silicon modulator by employing the double slots configuration with hybrid plasmonic effects. Two modulators, i.e., metal-insulator-metal and insulator-metal-insulator configurations have been demonstrated, showing that the double slots design can significantly improve the modulation efficiency. The obtained modulation efficiency is up to 0.525 dB/μm per graphene layer, far exceeding previous studies. It can be found that the light-graphene interaction plays a pivotal role in the modulation efficiency, whereas the height of metal has profound influence on the modulation. Our results may promote various future modulation devices based on graphene.