Gauge theory and thermalization are both topics of essential importance for modern quantum science and technology. The recently realized atomic quantum simulator for lattice gauge theories provides a unique opportunity for studying thermalization in gauge theory, in which theoretical studies have shown that quantum thermalization can signal the quantum phase transition. Nevertheless, the experimental study remains a challenge to accurately determine the critical point and controllably explore the thermalization dynamics due to the lack of techniques for locally manipulating and detecting matter and gauge fields. We report an experimental investigation of the quantum criticality in the lattice gauge theory from both equilibrium and nonequilibrium thermalization perspectives, with the help of the single-site addressing and atom-number-resolved detection capabilities. We accurately determine the quantum critical point and observe that the Néel state thermalizes only in the critical regime. This result manifests the interplay between quantum many-body scars, quantum criticality, and symmetry breaking.