The cytotoxic activity of several serotonin transporter (SERT) inhibitors and subtype of serotonin receptor 1A (5-HT1A receptor) ligands have been examined in androgen-insensitive human PC-3 prostate and neuroblastoma SH-SY5Y cancer cells. Almost all of the studied compounds (except 5-HT1A receptor agonist (2R)-(+)-8-Hydroxy-2-(di-n-propylamino)tetralin hydrobromide (8-OH-DPAT)) exhibited absolute cytotoxic activity against the examined cancer cells. The compound 4-Fluoro-N-[2-[4-(7-methoxy-1-naphthalenyl)-1-piperazinyl]ethyl]benzamide hydrochloride (S14506) that showed highest activity against neuroblastoma tumors was the 5-HT1A receptor agonist (although not alike other 5-HT1A receptor agonists). On the other hand, the compound 6-nitro-2-(4-undecylpiperazin-1-yl)quinoline hydrochloride (AZ07) that had the highest activity against PC-3 prostate cancer cells was a compound exhibiting antagonistic activity against the 5-HT1A receptor. Thus, compounds of oncotoxic properties S14506 and AZ07 should be evaluated further for their potential use in the prevention and treatment of cancer. Most of the 15 compounds tested exhibited either agonistic or antagonistic activity for both the cyclic adenosine monophosphate (cAMP) and extracellular signal-regulated kinase 1 and 2 (ERK1/2) pathways in human embryonic kidney 293 (HEK293) cells that overexpress the 5HT1AR gene. However, compounds paroxetine, N-Ac-paroxetine and 2-[4-(cyclobutylmethyl)piperazin-1-yl]-6-nitroquinoline hydrochloride (AB22) simultaneously exhibited antagonistic activity on the cAMP pathway and agonistic activity on the ERK1/2 pathway. Fluoxetine relative to compound AZ07 had almost three times lower cytotoxic activity against PC-3 prostate cancer cells. However, the proapoptotic activity of fluoxetine compared to compound AZ07 is almost two times higher which would suggest that the cytotoxic activity of both compounds may be dependent on different cell death mechanisms. Compound S14506 was found to be an antagonist of the serine-threonine protein kinase B (Akt) pathway. Prosurvival Akt activity may be reversed by Akt antagonists. Therefore, the antagonistic activity of S14506 on the Akt pathway may evoke caspase-3 expression and cytotoxicity. It appears that one should not expect a straightforward relationship between the activation of particular serotonergic pathways by selective serotonin reuptake inhibitors (SSRIs) and 5-HT1A receptor ligands and their cytotoxic or cytoprotective activity. Additionally, nuclear transcription factor κB (NF-κB), which may be involved in 5-HT-dependent biochemical pathways by coordinating different subunits in the formation of a dimer, may regulate the transcription of different transduction pathways. Therefore, it can be suggested that the mechanism of the cytotoxic activity of certain compounds (serotonergic against nonserotonergic) may depend on the compound and cancer type being examined. Docking studies showed that S14506, buspirone and spiperone bind in similar ways in the 5-HT1A receptor model and interacted with similar 5-HT1A receptor residues. S14506 and spiperone were found to be located closer to both phenylalanines in TM6 than buspirone, thus exhibiting more antagonist binding modes.
Keywords: 5-HT1A receptor; PC-3; S14506; SH-SY5Y; SSRI; apoptosis; cancer; cytotoxic activity; serotonin ligands.