Protein O-linked β-N-acetylglucosamine (O-GlcNAc) modification (O-GlcNAcylation) is a unique monosaccharide modification discovered in the early 1980s. With the technological advances in the past several decades, great progress has been made to reveal the biochemistry of O-GlcNAcylation, the substrates of O-GlcNAcylation, and the functional importance of protein O-GlcNAcylation. As a nutrient sensor, protein O-GlcNAcylation plays important roles in almost all biochemical processes examined. Although the functional importance of O-GlcNAcylation of proteins has been extensively reviewed previously, the chemical and biochemical aspects have not been fully addressed. In this review, by critically evaluating key publications in the past 35 years, we aim to provide a comprehensive understanding of this important post-translational modification (PTM) from analytical and biochemical perspectives. Specifically, we will cover (1) multiple analytical advances in the characterization of O-GlcNAc cycling components (i.e., the substrate donor UDP-GlcNAc, the two key enzymes O-GlcNAc transferase and O-GlcNAcase, and O-GlcNAc substrate proteins), (2) the biochemical characterization of the enzymes with a variety of chemical tools, and (3) exploration of O-GlcNAc cycling and its modulating chemicals as potential biomarkers and therapeutic drugs for diseases. Last but not least, we will discuss the challenges and possible solutions for basic and translational research of protein O-GlcNAcylation in the future.