We investigated the evolution of ferromagnetism in layered Fe3GeTe2 flakes under different pressures and temperatures using in situ magnetic circular dichroism (MCD) spectroscopy. We found that the rectangular shape of the hysteresis loop under an out-of-plane magnetic field sweep can be sustained below 7 GPa. Above that pressure, an intermediate state appears in the low-temperature region signaled by an 8-shaped skewed hysteresis loop. Meanwhile, the coercive field and Curie temperature decrease with increasing pressures, implying the decrease of the exchange interaction and the magneto-crystalline anisotropy under pressures. The intermediate phase has a labyrinthine domain structure, which is attributed to the increase of the ratio of exchange interaction to magneto-crystalline anisotropy based on Jagla's theory. Moreover, our calculations reveal a weak structural transition around 6 GPa that corresponds to a significant change in the FeI-FeI bond length, which has strong influences on magnetic interaction. Detailed analysis on exchange interaction and magneto-crystalline anisotropy with pressure shows a consistent trend with experiments.