Graphene is an optical material of unusual characteristics because of its linearly dispersive conduction and valence bands and the strong interband transitions. It allows broadband light-matter interactions with ultrafast responses and can be readily pasted to surfaces of functional structures for photonic and optoelectronic applications. Recently, graphene-based optical modulators have been demonstrated with electrical tuning of the Fermi level of graphene. Their operation bandwidth, however, was limited to about 1 GHz by the response of the driving electrical circuit. Clearly, this can be improved by an all-optical approach. Here, we show that a graphene-clad microfiber all-optical modulator can achieve a modulation depth of 38% and a response time of ∼ 2.2 ps, limited only by the intrinsic carrier relaxation time of graphene. This modulator is compatible with current high-speed fiber-optic communication networks and may open the door to meet future demand of ultrafast optical signal processing.