We investigate the time-averaged square displacement (TASD) of continuous-time random walks with respect to the number of steps N which the random walker performed during the data acquisition time T. We prove that in each realization the TASD grows asymptotically linear in the lag time τ and in N, provided the steps cannot accumulate in small intervals. Consequently, the fluctuations of the latter are dominated by the fluctuations of N, and fluctuations of the walker's thermal history are irrelevant. Furthermore, we show that the relative scatter decays as 1/sqrt[N], which suppresses all nonlinear features in a plot of the TASD against the lag time. Parts of our arguments also hold for continuous-time random walks with correlated steps or with correlated waiting times.