Lupulone, a β-acid derived from hop extracts has been shown to exhibit cytotoxic activity against cancer cells. In this study we investigated the functional role of different modes of cell death that mediate anticancer effect of lupulone derivatives in prostate cancer cells. ELISA, immunoblotting and siRNA approaches were utilised to study cell death, expression of proteins of interest and their functional activities. We found that the anticancer effect of lupulone derivatives on prostate cancer cells is associated with induction of apoptosis and autophagy as determined by increases of DNA fragmentation and LC3I/ LC3II conversion respectively. Inhibition of apoptosis using a pan-caspase inhibitor resulted in increased levels of autophagy. Following screening of proteins associated with autophagy we found that Atg4β expression was increased in prostate cancer cells after treatment with lupulone. Transfection of cells with siRNA against Atg4β resulted in increased levels of apoptosis in prostate cancer cells. Treatment of prostate cancer cells with lupulone derivatives initiated two modes of cell death: apoptosis as a killing pathway and autophagy as a protection against cell death. Further studies are required to investigate the regulation of Atg4β activity in lupulone derivatives-induced negative crosstalk between apoptosis and autophagy.