We derive a variational expression for the correlation time of physical observables in steady-state diffusive systems. As a consequence of this variational expression, we obtain lower bounds on the correlation time, which provide speed limits on the self-averaging of observables. In equilibrium, the bound takes the form of a trade-off relation between the long- and short-time fluctuations of an observable. Out of equilibrium, the trade-off can be violated, leading to an acceleration of self-averaging. We relate this violation to the steady-state entropy production rate, as well as the geometric structure of the irreversible currents, giving rise to two complementary speed limits. One of these can be formulated as a lower estimate on the entropy production from the measurement of time-symmetric observables. Using an illustrating example, we show the intricate behavior of the correlation time out of equilibrium for different classes of observables and how this can be used to partially infer dissipation even if no time-reversal symmetry breaking can be observed in the trajectories of the observable.