We investigate the transport and localization of tracer probes in a glassy matrix as a function of relative size using dynamic X-ray scattering experiments and molecular dynamics simulations. The quiescent relaxations of tracer particles evolve with increasing waiting time, tw. The corresponding relaxation times increase exponentially at small tw and then transition to a power-law behavior at longer tw. As tracer size decreases, the aging behavior weakens and the particles become less localized within the matrix until they delocalize at a critical size ratio δ0 ≈ 0.38. Localization does not vary with sample age even as the relaxations slow by approximately an order of magnitude, suggesting that matrix structure controls tracer localization.