The aim of this study was to investigate the effect of short hairpin RNA (shRNA) targeting autotaxin (ATX) on the migratory and invasive capability of AGS human gastric cancer cells and the growth of xenografts in nude mice. pSUPER-ATX and pSUPER-mock (non-specific), which were constructed corresponding to the ATX-shRNA and negative control mock-shRNA synthesized based on gene sequence, were transfected with blank plasmid pSUPER-control into AGS human gastric cancer cells using Lipofectamine. At 24, 48 and 72 h post-transfection, the cells were harvested and analyzed. The endogenous ATX mRNA and protein of the different groups of AGS cells were detected by RT-PCR and western blot assays. Cell proliferation was measured by MTT assay. In vitro Transwell and Matrigel assays were used to detect the cell migratory and invasive capabilities. A tumor xenograft model was generated by subcutaneous injection of AGS cells into the dorsum of nude mice. The growth of xenograft tumors was monitored and measured; changes in tumor morphology and the organs of mice were observed by H&E staining. The expression of ATX, MMP-2 and MMP-9 protein in xenograft tumor tissues was detected by immunohistochemistry and western blotting. In vitro, both the mRNA and protein levels of ATX in the pSUPER-ATX group were significantly downregulated (P<0.01), and the cell proliferative, migratory and invasive capabilities were also significantly decreased. In vivo, no obvious damage to the organs was found. pSUPER-ATX significantly suppressed the tumor volume and weight from the 7th week after cell transplantation, compared to the pSUPER-mock, pSUPER-control and WT groups; the inhibition rates were approximately 50% (P<0.05). However, no significant differences in these parameters were found among the WT, pSUPER-mock and pSUPER-control groups. Furthermore, ATX, MMP-2 and MMP-9 were expressed at significantly lower levels in the pSUPER-ATX group compared to the levels in the other three groups (P<0.05), and a significant correlation between ATX and MMP-2 expression was found (r=0.869, P<0.01). The specific shRNA targeting ATX downregulated the endogenous ATX of AGS human gastric cancer cells, and inhibited AGS cell proliferative, migratory and invasive potentials. Moreover, shRNA targeting ATX inhibited the growth of human gastric cancer xenografts in nude mice.