We present the results of direct interferometric measurements on the pulse-to-pulse phase jitter of a metrological, fiber-based, infrared (IR) frequency comb. We show that the short-time evolution of such phase fluctuations, which cannot be actively controlled by any feedback system, imposes a stringent limit on the tooth linewidth of extreme ultraviolet (XUV) combs produced by high-order harmonic conversion, thus explaining the difference of 9 orders of magnitude between the coherence times of state-of-the-art IR and XUV frequency combs.