We report on direct observation of temporal contrast degradation of short pulses amplified by optical parametric chirped-pulse amplification. We show that, despite injection seeding, quantum-noise-induced fast modulations (< 50 ps) of the temporal profile of the pump pulse are imprinted on the spectrum of the amplified chirped pulse and give rise to a large picosecond pedestal in the time domain.