Gravitational waves from a source moving relative to us can suffer from special-relativistic effects such as aberration. The required velocities for these to be significant are on the order of 1000 km s^{-1}. This value corresponds to the velocity dispersion that one finds in clusters of galaxies. Hence, we expect a large number of gravitational-wave sources to have such effects imprinted in their signals. In particular, the signal from a moving source will have its higher modes excited, i.e., (3,3) and beyond. We derive expressions describing this effect and study its measurability for the specific case of a circular, nonspinning extreme-mass-ratio inspiral. We find that the excitation of higher modes by a peculiar velocity of 1000 km s^{-1} is detectable for such inspirals with signal-to-noise ratios of ≳20. Using a Fisher matrix analysis, we show that the velocity of the source can be measured to a precision of just a few percent for a signal-to-noise ratio of 100. If the motion of the source is ignored, parameter estimates could be biased, e.g., the estimated masses of the components through a Doppler shift. Conversely, by including this effect in waveform models, we could measure the velocity dispersion of clusters of galaxies at distances inaccessible to light.