The present study investigated the effects of formononetin (FMN) against balloon injury‑induced neointima formation in vivo and platelet‑derived growth factor (PDGF)‑BB‑induced proliferation and migration of vascular smooth muscle cells (VSMCs) in vitro, and explored the underlying mechanisms. A rat model of carotid artery injury was established, in order to examine the effects of FMN on balloon injury‑induced neointima formation. Histological observation of the carotid artery tissues was conducted by hematoxylin and eosin staining. VSMC proliferation during neointima formation was observed by proliferating cell nuclear antigen staining. Subsequently, rat aortic VSMCs were isolated, and the effects of FMN on PDGF‑BB‑induced VSMC proliferation and migration were determined using Cell Counting Kit‑8 and Transwell/wound healing assays, respectively. Immunohistochemical and immunocytochemical staining was applied to measure the expression of transforming growth factor (TGF)‑β in carotid artery tissues and VSMCs, respectively. SMAD family member 3 (Smad3)/phosphorylated (p)‑Smad3 expression was examined by western blotting. FMN treatment significantly inhibited the abnormal proliferation of smooth muscle cells in neointima, and alterations to the vascular structure were attenuated. In addition, pretreatment with FMN effectively inhibited the proliferation of PDGF‑BB‑stimulated VSMCs (P<0.05). FMN also reduced the number of cells that migrated to the lower surface of the Transwell chamber and decreased wound‑healing percentage (P<0.05). The expression levels of TGF‑β were decreased by FMN treatment in vivo and in vitro, and Smad3/p‑Smad3 expression was also markedly inhibited. In conclusion, FMN significantly protected against balloon injury‑induced neointima formation in the carotid artery of a rat model; this effect may be associated with the regulation of VSMC proliferation and migration through altered TGF‑β1/Smad3 signaling.