Ruthenium-based compounds have gained great interest due to their potent cytotoxicity in cancer cells; however, much of their potential applications remain unexplored. In this paper, we report the synthesis of a novel ruthenium complex with xanthoxylin (RCX) and the investigation of its cellular and molecular action in human hepatocellular carcinoma HepG2 cells. We found that RCX exhibited a potent cytotoxic effect in a panel of cancer cell lines in monolayer cultures and in a 3D model of multicellular cancer spheroids formed from HepG2 cells. This compound is detected at a high concentration in the cell nuclei, induces DNA intercalation and inhibits DNA synthesis, arresting the cell cycle in the S-phase, which is followed by the activation of the caspase-mediated apoptosis pathway in HepG2 cells. Gene expression analysis revealed changes in the expression of genes related to cell cycle control, apoptosis and the MAPK pathway. In addition, RCX induced the phosphorylation of ERK1/2, and pretreatment with U-0126, an MEK inhibitor known to inhibit the activation of ERK1/2, prevented RCX-induced apoptosis. In contrast, pretreatment with a p53 inhibitor (cyclic pifithrin-α) did not prevent RCX-induced apoptosis, indicating the activation of a p53-independent apoptosis pathway. RCX also presented a potent in vivo antitumor effect in C.B-17 SCID mice engrafted with HepG2 cells. Altogether, these results indicate that RCX is a novel anticancer drug candidate.