Single Nucleotide Polymorphic (SNP) variations of proinflammatory cytokines such as Tumor Necrosis Factor-α (TNF-α) have been reported to be closely associated with the major depressive disorder (MDD). However, it is unclear if proinflammatory genetic burden adversely affects the regional gray matter volume in patients with MDD. The aim of this study was to test whether rs1799724, an SNP of TNF-α, contributes to the neuroanatomical changes in MDD. In this cross-sectional study, a total of 144 MDD patients and 111 healthy controls (HC) well matched for age, sex and education were recruited from Shanghai Mental Health Center. Voxel-based morphometry (VBM) followed by graph theory based structural covariance analysis was applied to locate diagnosis x genotype interactions. Irrespective of diagnosis, individuals with the high-risk genotype (T-carriers) had reduced volume in left angular gyrus (main effect of genotype). Diagnosis x genotype interaction was exclusively localized to the visual cortex (right superior occipital gyrus). The same region also showed reduced volume in patients with MDD than HC (main effect of diagnosis), with this effect being most pronounced in patients carrying the high-risk genotype. However, neither global nor regional network of structural covariance was found to have group difference. In conclusion, a genetic variation which can increase TNF-α expression selectively affects the anatomy of the visual cortex among the depressed subjects, with no effect on the topographical organization of multiple cortical regions. This supports the notion that anatomical changes in depression are in part influenced by the genetic determinants of inflammatory activity.