Analyzing diverse seismic catalogs, we have determined that the probability densities of the earthquake recurrence times for different spatial areas and magnitude ranges can be described by a unique universal distribution if the time is rescaled with the rate of seismic occurrence, which therefore fully governs seismicity. The shape of the distribution shows the existence of clustering beyond the duration of aftershock bursts, and scaling reveals the self-similarity of the clustering structure in the space-time-magnitude domain. This holds from worldwide to local scales, for quite different tectonic environments and for all the magnitude ranges considered.