We investigate ergodic time scales in single-particle tracking by introducing a covariance measure Ω(Δ;t) for the time-averaged relative square displacement recorded in lag-time Δ at elapsed time t. The present model is established in the generalized Langevin equation with a power-law memory function. The ratio Ω(Δ;Δ)/Ω(Δ;t) is shown to obey a universal scaling law for long but finite times and is used to extract the effective ergodic time. We derive a finite-time-averaged Green-Kubo relation and find that, to control the deviations in measurement results from ensemble averages, the ratio Δ/t must be neither too small nor close to unity. Our paper connects the experimental self-averaging property of a tracer with the theoretic velocity autocorrelation function and sheds light on the transition to ergodicity.