The aim of the present study was to investigate the anti‑tumor effect of apogossypolone (ApoG2) on human LNCaP cells in vitro and in vivo. Cell viability was evaluated using an MTT assay. Cell autophagy and apoptosis were detected by flow cytometry and using a terminal deoxynucleotidyl transferase dUTP nick end labeling assay, respectively. Morphological autophagy alterations were observed by transmission electron microscopy. The formation of acidic vesicular organelles was assessed by acridine orange staining and fluorescence microscopy. Quantitative polymerase chain reaction (qPCR) was conducted to detect the expression levels of apoptosis‑associated protein B‑cell lymphoma 2 (Bcl‑2) and Bak. The models of transplantation tumors in nude mice were established via subcutaneous injection of LNCaP cells. Growth of LNCaP cells was inhibited by ApoG2 treatment. Flow cytometry demonstrated that ApoG2 induced apoptosis in LNCaP cells. The Bcl‑2 expression was decreased while Bak expression was increased. In addition, activation of cysteine aspartate protease (caspase)‑3 and ‑8 was observed and 3‑methyladenine (3‑MA) enhanced apoptosis of LNCaP cells. Furthermore, nude mice treated with ApoG2 demonstrated a significant decrease in tumor volume and a significant increase in cell viability. Immunohistochemical analysis of tumor tissues demonstrated that ApoG2 enhanced caspase‑3, ‑8, LC‑3B and beclin‑1 expression and reduced the expression of Bcl‑2. ApoG2 was able to effectively suppress the growth of LNCaP cells through the induction of autophagy and apoptosis.