We present a study of all-optical light manipulation arising in a graphene-embedded side-polished fiber (SPF) with a Norland Optical Adhesives (NOA)-coated structure. With the help of the Pauli blocking effect, such an all-fiber device serves to manage the loss of transverse-electric-polarized light when the control light and the signal light are polarized along the direction parallel to the graphene surface. The insertion loss of this device can be effectively reduced with the NOA coating. An enhanced interaction between the graphene and the propagated light can be achieved via the strong evanescent field of the SPF and longer interaction length. This results in effective all-optical manipulation of light with a modulation depth of 10.4 dB (or modulation efficiency of ∼91%) and a modulation slope of ∼1.3, where the required control power is only about 14 dBm. The device has broadband operation wavelength. The insertion loss for both the signal light and the control light are only about 0.6 dB. The experimental results are well-fitting with the simulation study. Such an all-fiber device has the potential for all-optical signal processing.