Although it has been known for half a century that the physical aging of glasses in experiments is described well by a linear thermal-history convolution integral over the so-called material time, the microscopic definition and interpretation of the material time remains a mystery. We propose that the material-time increase over a given time interval reflects the distance traveled by the system's particles. Different possible distance measures are discussed, starting from the standard mean-square displacement and its inherent-state version that excludes the vibrational contribution. The viewpoint adopted, which is inspired by and closely related to pioneering works of Cugliandolo and Kurchan from the 1990s, implies a "geometric reversibility" and a "unique-triangle property" characterizing the system's path in configuration space during aging. Both of these properties are inherited from equilibrium, and they are here confirmed by computer simulations of an aging binary Lennard-Jones system. Our simulations moreover show that the slow particles control the material time. This motivates a "dynamic-rigidity-percolation" picture of physical aging. The numerical data show that the material time is dominated by the slowest particles' inherent mean-square displacement, which is conveniently quantified by the inherent harmonic mean-square displacement. This distance measure collapses data for potential-energy aging well in the sense that the normalized relaxation functions following different temperature jumps are almost the same function of the material time. Finally, the standard Tool-Narayanaswamy linear material-time convolution-integral description of physical aging is derived from the assumption that when time is replaced by distance in the above sense, an aging system is described by the same expression as that of linear-response theory.