Applications with optical atomic clocks and precision timing often require the transfer of optical frequency references to the electrical domain with extremely high fidelity. Here we examine the impact of photocarrier scattering and distributed absorption on the photocurrent noise of high-speed photodiodes when detecting ultralow jitter optical pulses. Despite its small contribution to the total photocurrent, this excess noise can determine the phase noise and timing jitter of microwave signals generated by detecting ultrashort optical pulses. A Monte Carlo simulation of the photodetection process is used to quantitatively estimate the excess noise. Simulated phase noise on the 10 GHz harmonic of a photodetected pulse train shows good agreement with previous experimental data, leading to the conclusion that the lowest phase noise photonically generated microwave signals are limited by photocarrier scattering well above the quantum limit of the optical pulse train.