Two types of activated carbons have been prepared by H3PO4 activation of lignocellulose and by H3PO4 modification of activated carbon, and then heat-treated at temperatures from 400 to 900 °C in an atmosphere of N2 or H2 to investigate the evolution of phosphorus-containing groups. Elemental analysis, X-ray photoelectron spectroscopy, 31P nuclear magnetic resonance, nitrogen adsorption, and scanning electron microscopy have been used to analyze the physicochemical properties of the activated carbons. The results show that C-O-P linkages of phosphorus-containing groups can progressively evolve into C-P-O, C3-P═O, C3-P, and eventually elemental phosphorus as a result of heat treatment. Phosphate-like groups are much more thermally stable in an N2 than in an H2 atmosphere. In N2, C-O-P linkages significantly evolve into C-P-O and C3-P═O at up to 800 °C, whereas C3-P linkages are not formed even at 900 °C. In H2, the corresponding evolution remarkably occurs at 500 °C, forming C3-P linkages and eventually elemental phosphorus. Moreover, the two activated carbons exhibit different evolution trends, suggesting that the evolution happens more easily for phosphorus-containing groups located on the edges of graphite-like crystallites than those in the lattice. Finally, we propose different evolution pathways of phosphorus-containing groups upon heat treatment in N2 and H2 atmospheres.