We present an experimental and numerical study on the spectrally resolved pump-to-output intensity noise coupling in soliton fiber oscillators. In our study, we observe a strong pump noise coupling to the Kelly sidebands, while the coupling to the soliton pulse is damped. This behavior is observed in erbium-doped as well as holmium-doped fiber oscillators and confirmed by numerical modeling. It can be seen as a general feature of laser oscillators in which soliton pulse formation is dominant. We show that spectral blocking of the Kelly sidebands outside the laser cavity can improve the intensity noise performance of the laser dramatically.