In the present work, change in speckle-pattern of linearly polarized light passed through graphene-covered optical fiber placed in external magnetic field is investigated. The possibility of magnetic speckle-pattern rotation suppression and inverse speckle-pattern rotation effect is shown. This effect can be controlled by a chemical potential of graphene layer, which can be changed easily by a gate voltage, for example. For quartz optical fiber at wavelength 0.633 μm, core diameter 9 μm, and fiber length 5 cm, an inverse rotation value of 17° is reached at chemical potential of graphene layer about 1 eV and magnetic field strength 30 kOe. Results of the work may be useful for different magneto-optics, opto-electronics, and photonics applications.