The effect of esculetin, a coumarin derivative with lipoxygenase inhibitor activity, on the proliferation response of cultured rabbit vascular smooth muscle cells was studied. Proliferation response was determined by the uptake of tritiated thymidine. Esculetin (10(-5)-10(-4) M) dose dependently inhibited the enhanced proliferation stimulated by 5% fetal calf serum. The structure-activity relationship of esculetin and eight other coumarin derivatives indicates that two adjacent phenolic hydroxyl groups at the C-6 and C-7 positions in the coumarin skeleton are necessary for the potent antiproliferative effect. The antiproliferative effects of other lipoxygenase inhibitors, 5,8,11,14-eicosatetraynoic acid (ETYA) and ketoconazole, were comparable to the effect of esculetin. However, esculetin exhibited the greatest maximal suppression. The enhanced releases of 12-hydroxyeicosatetraenoic acid (12-HETE), prostaglandin E2 and 6-keto-prostaglandin F1 alpha in the culture medium of smooth muscle cells stimulated by 5% fetal calf serum were significantly reduced by esculetin. Furthermore, the fetal calf serum-stimulated protein tyrosine kinase activity was reduced by esculetin (10(-5)-10(-4) M) in a dose-dependent manner. In contrast, the protein kinase C activity stimulated by phorbol-12-myristate-13-acetate was not affected by esculetin (10(-6)-10(-4) M). These results suggest that the antiproliferative effect of esculetin on vascular smooth muscle cells may be partly mediated through inhibition of protein tyrosine kinase and modulated by inhibition of lipoxygenase.