Given the ubiquity of glass formulations that are functionalized with silver compounds, the electronic interaction between isolated silver cations and the glass network deserves more attention. Here, we report the structural origin of the optical properties that result from silver doping in fluorophosphate (PF) and sulfophosphate (PS) glasses. To achieve this, solid-state nuclear magnetic resonance (NMR) spectroscopy and density functional theory (DFT) are combined with optical spectroscopic analysis and physical property measurements. Comparing the 31P NMR, 27Al 1d NMR, and 27Al multi-quantum magic-angle spinning NMR of doped glasses and glasses with large amounts of Ag+ added, we deduce silver's bonding preference in these mixed-anion aluminophosphate glasses. We show that such understanding provides an explanation for the large Stokes shift observed for Ag+ in PF and PS glasses, which is related to absorption by the ionic Ag+···-O-P species and transfer of the excitation energy within more covalently bonded Ag2O-like clusters. This is corroborated by DFT calculations, which show that the Ag+···-O-P and Ag+···-O-S bonds in corresponding crystals are mostly ionic. The introduction of more silver ions into the crystal structure results in more covalent bonding between Ag+ and the phosphate matrix.