We investigate the effect of coexisting transverse modes on the operation of self-mixing sensors based on vertical-cavity surface-emitting lasers (VCSELs). The effect of multiple transverse modes on the measurement of displacement and distance were examined by simulation and in laboratory experiment. The simulation model shows that the periodic change in the shape and magnitude of the self-mixing signal with modulation current can be properly explained by the different frequency-modulation coefficients of the respective transverse modes in VCSELs. The simulation results are in excellent agreement with measurements performed on single-mode and multimode VCSELs and on self-mixing sensors based on these VCSELs.