The rotational Doppler effect (RDE) of optical vortex beams provides an efficient way to measure the rotational frequency of objects based on rotational Doppler frequency shifts, while the frequency shift signals display a distinct broadening effect when the vortex beam is laterally misaligned with the center of rotation of a planar object. We use a modal decomposition method to reveal the broadening effect and obtain a linear fitting equation between the quantity of signals and lateral misalignments. In an experiment of RDE, the lateral misalignment is extracted from the quantity of signal peaks. The fitting equation is proved to be precise within the uncertainty of ±0.17mm (±2.8% of the vortex beam radius), and the center of rotation is located with an error less than 3.33% of the beam radius. Our work provides a new approach to locate the center of rotation of noncooperative objects, which may be valuable in mechanical manufacturing and optical noncontact metrology.