ADAMTS9 belongs to the ADAMTS (a disintegrin and metalloproteinase with thrombospondin motifs) protein family, and its expression is frequently silenced due to promoter hypermethylation in various human cancers. However, the underlying mechanisms remain largely unknown. In this study, we investigated the inhibitory effects of ADAMTS9 on gastric cancer (GC) cells. We initially examined ADAMTS9 protein level in 135 GC and adjacent normal tissue pairs, showing that ADAMTS9 was strikingly decreased in the malignant specimens and patients with low ADAMTS9 expression exhibited more malignant phenotypes and poorer outcome. ADAMTS9 expression was restored in AGS and BGC-823 cells, which then markedly suppressed cellular viability and motility in vitro and in vivo. As ADAMTS9 was enriched in the nuclei of gastric mucosal cells, RNA-sequencing experiment showed that ADAMTS9 significantly altered gene expression profile in BGC-823 cells. Additionally, DNA methyltransferase 3α (DNMT3A) was identified to be responsible for the hypermethylation of ADAMTS9 promoter, and this methyltransferase was ubiquitinated by ring finger protein 180 (RNF180) and then subject to proteasome-mediated degradation. In conclusion, we uncovered RNF180/DNMT3A/ADAMTS9 axis in GC cells and showed how the signaling pathway affected GC cells.