Nonlinear optical applications depend on pulse duration and coherence of the laser pulses. Characterization of high-repetition rate pulsed laser sources can be complicated by their pulse-to-pulse instabilities. Here, we introduce and demonstrate experimentally a quantitative measurement that can be used to determine the pulse-to-pulse fidelity of ultrafast laser sources. Numerical simulations and experiments illustrate the effect of spectral phase and amplitude noise on second and third harmonic generation.