The extremely low loss of silica fibers has enabled the telecommunication revolution, but single-mode fiber-optic communication systems have been driven to their capacity limits. As a means to overcome this capacity crunch, space-division multiplexing (SDM) using few-mode fibers (FMF) has been proposed and demonstrated. In single-mode optical fibers, Rayleigh scattering serves as the dominant mechanism for optical loss. However, to date, the role of Rayleigh scattering in FMFs remains elusive. Here we establish and experimentally validate a general model for Rayleigh scattering in FMFs. Rayleigh backscattering not only sets the intrinsic loss limit for FMFs but also provides the theoretical foundation for few-mode optical time-domain reflectometry, which can be used to probe perturbation-induced mode-coupling dynamics in FMFs. We also show that forward inter-modal Rayleigh scattering ultimately sets a fundamental limit on inter-modal-crosstalk for FMFs. Therefore, this work not only has implications specifically for SDM systems but also broadly for few-mode fiber optics and its applications in amplifiers, lasers, and sensors in which inter-modal crosstalk imposes a fundamental performance limitation.