The electron-phonon interaction is well known to create major resistance to electron transport in metals and semiconductors, whereas fewer studies are directed to its effect on phonon transport, especially in semiconductors. We calculate the phonon lifetimes due to scattering with electrons (or holes), combine them with the intrinsic lifetimes due to the anharmonic phonon-phonon interaction, all from first principles, and evaluate the effect of the electron-phonon interaction on the lattice thermal conductivity of silicon. Unexpectedly, we find a significant reduction of the lattice thermal conductivity at room temperature as the carrier concentration goes above 10(19) cm(-3) (the reduction reaches up to 45% in p-type silicon at around 10(21) cm(-3)), a range of great technological relevance to thermoelectric materials.