Lupulone, a β-acid derived from hop extracts has been shown to exhibit antibacterial and anticancer activity. In this study we investigated the anticancer potency of lupulone and its novel derivatives and their mechanism of action on prostate cancer cells. Cell viability was determined using the MTT assay, and the ELISA approach was used to investigate induction of apoptosis. Immunoblot analysis was carried out to determine activation and regulation of proteins associated with cell death. Screening of natural and new lupulone derivatives for their anticancer activity demonstrated that one (lupulone derivative 1h) displayed stronger anticancer activity than lupulone itself on PC3 and DU145 prostate cancer cells. We further found that lupulone derivatives induced caspase-dependent apoptosis that is associated with activation of caspases 8, 9, and 3. Furthermore, caspase 8 inhibitor Z-IETD-fmk reduced cell death induced by lupulone derivatives, suggesting that apoptosis is mediated by caspase 8. Finally, we found that lupulone and its synthetic derivatives also increased formation of LC3II suggesting that autophagy is also implicated in prostate cancer cell death. The new lupulone derivatives induce caspase-dependent apoptosis and autophagy in prostate cancer cells and appear to be good candidates for further preclinical studies of prostate cancer treatment.