The aim of this study was to search for new antiviral agents from Chinese herbal medicine. Pure flavonoids and aqueous extracts of Caesalpinia pulcherrima Swartz were used in experiments to test their influence on a series of viruses, namely herpesviruses (HSV-1, HSV-2) and adenoviruses (ADV-3, ADV-8, ADV-11). The EC50 was defined as the concentration required to achieve 50% protection against virus-induced cytopathic effects, and the selectivity index (SI) was determined as the ratio of CC50 (concentration of 50% cellular cytotoxicity) to EC50. Results showed that aqueous extracts of C. pulcherrima and its related quercetin possessed a broad-spectrum antiviral activity. Among them, the strongest activities against ADV-8 were fruit and seed (EC50 = 41.2 mg/l, SI = 83.2), stem and leaf (EC50 = 61.8 mg/l, SI = 52.1) and flower (EC50 = 177.9 mg/l, SI = 15.5), whereas quercetin possessed the strongest anti-ADV-3 activity (EC50 = 24.3 mg/l, SI = 20.4). In conclusion, some compounds of C. pulcherrima which possess antiviral activities may be derived from the flavonoid of quercetin. The mode of action of quercetin against HSV-1 and ADV-3 was found to be at the early stage of multiplication and with SI values greater than 20, suggesting the potential use of this compound for treatment of the infection caused by these two viruses.