Further development of GaN/AlGaN based optoelectronic devices requires optimization of the p-type material growth process. In particular, uncontrolled diffusion of Mg dopants may decrease the performance of a device. Thus it is meaningful to study the behavior of Mg and the origins of its diffusion in detail. In this work we have employed secondary ion mass spectrometry to study the diffusion of magnesium in GaN/AlGaN structures. We show that magnesium has a strong tendency to form Mg-H complexes which immobilize Mg atoms and restrain their diffusion. However, these complexes are not present in samples post-growth annealed in an oxygen atmosphere or Al-rich AlGaN structures which naturally have a high oxygen concentration. In these samples, more Mg atoms are free to diffuse and thus the average diffusion length is considerably larger than for a sample annealed in an inert atmosphere.