Two new scalarane sesterterpenoids, 12β-(3'β-hydroxybutanoyloxy)-20,24-dimethyl-24-oxo-scalara-16-en-25-al (1) and 12β-(3'β-hydroxypentanoyloxy)-20,24-dimethyl-24-oxo-scalara-16-en-25-al (2), along with one known tetraprenyltoluquinol-related metabolite (3), were isolated from the sponge Carteriospongia sp. In leukemia Molt 4 cells, 1 at 0.0625 μg/mL (125 nM) triggered mitochondrial membrane potential (MMP) disruption and apoptosis showing more potent effect than 2 and 3. The isolates inhibited topoisomerase IIα expression. The apoptotic-inducing effect of 3 was supported by the in vivo experiment through suppressing the volume of xenograft tumor growth (47.58%) compared with the control. Compound 1 apoptotic mechanism of action in Molt 4 cells was further elucidated through inducing ROS generation, calcium release and ER stress. Using the molecular docking analysis, 1 exhibited more binding affinity to N-terminal ATP-binding pocket of Hsp90 protein than 17-AAG, a standard Hsp90 inhibitor. The expression of Hsp90 client proteins, Akt, p70S6k, NFκB, Raf-1, p-GSK3β, and XIAP, MDM 2 and Rb2, and CDK4 and Cyclin D3, HIF 1 and HSF1 were suppressed by the use of 1. However, the expression of Hsp70, acetylated tubulin, and activated caspase 3 were induced after 1 treatment. Our results suggested that the proapoptotic effect of the isolates is mediated through the inhibition of Hsp90 and topoisomerase activities.