Mesoscopic structures form in dense granular materials due to the self-organisation of the constituent particles. These structures have internal structural degrees of freedom in addition to the translational degree of freedom. The resultant granular elasticity, which exhibits intrinsic variations and inevitable relaxation, is a key quantity that accounts for macroscopic solid- or fluid-like properties and the transitions between them. In this work, we propose a potential energy landscape (PEL) with local stable basins and low elastic energy barriers to analyse the nature of granular elasticity. A function for the elastic energy density is proposed for stable states and is further calibrated with ultrasonic measurements. Fluctuations in the elastic energy due to the evolution of internal structures are proposed to describe a so-called configuration temperature T(c) as a counterpart of the classical kinetic granular temperature T(k) that is attributed to the translational degrees of freedom. The two granular temperatures are chosen as the state variables, and a fundamental equation is established to develop non-equilibrium thermodynamics for granular materials. Due to the relatively low elastic energy barrier in the PEL, granular elasticity relaxes more under common mechanical loadings, and a simple model based on mean-field theory is developed to account for this behaviour.